Intermediate everything in Analytics & Metrics, page 14
DAU/MAU Ratio: Measuring Product Stickiness
The DAU/MAU ratio measures product stickiness by comparing daily to monthly users. Social media apps aim for high ratios (50%+), while e-commerce expects lower ones. The footgun is comparing ratios without context; a 'good' number varies by product type.
Prophet: Automated Time Series Forecasting
Prophet treats forecasting like decomposing a signal, breaking a time series into trend, seasonality, and holiday effects. It excels at predicting business metrics with strong seasonal patterns, like web traffic.
Time Series Cross-Validation
Unlike standard cross-validation that shuffles data, time series cross-validation respects the arrow of time. It evaluates a forecasting model by training only on past data to predict a future point, then rolling forward. The footgun is using random k-folds.
ACF vs. PACF: A Time Series Signal and Its Echo
Autocorrelation (ACF) measures a time series's total correlation with its past, including indirect echoes. Partial Autocorrelation (PACF) isolates the direct correlation by removing intermediate effects. They help select forecasting model parameters.
Holt-Winters Exponential Smoothing
Forecast time-series data by giving more weight to recent events. Holt-Winters smoothing is used to analyze metrics and make predictions by accounting for user-defined assumptions like seasonality.
Exponential Smoothing: Weighting Recent Data More Heavily
Exponential smoothing creates forecasts from a weighted average of past data, where weights decay exponentially for older points. It's used for quick, reliable predictions like inventory demand. The footgun: simple versions ignore trends and seasonality.
Sample Ratio Mismatch (SRM): When Your A/B Test Is Broken
Sample Ratio Mismatch (SRM) means your A/B test's traffic split is broken, violating random assignment. For example, a 50/50 split results in a statistically significant imbalance.
The Novelty Effect: When New Isn't Always Better
The Novelty Effect is a temporary metric spike from a feature's newness, not its inherent value. It often appears in A/B tests for high-frequency products, inflating short-term metrics. The footgun is mistaking this initial excitement for a long-term win.
Average Treatment Effect (ATE): Isolating the Impact of a Change
The Average Treatment Effect (ATE) isolates an intervention's true impact by comparing the average outcome of a treated group to a control group. It's used in A/B tests and policy evaluations. The footgun is assuming causation without true randomization.
The Counterfactual Framework for Causal Inference
The Counterfactual Framework models causality by imagining two parallel universes for each person: one with a treatment, one without. It's the basis for A/B tests and analyzing observational data.
Standard Error: Gauging Your Measurement's Precision
Standard error measures the precision of a sample statistic, like the mean. It answers: "If I ran this experiment again, how much would my result change?" It's key for building confidence intervals and A/B testing. Don't confuse it with standard deviation.
Type I vs. Type II Errors: False Alarms vs. Missed Detections
A Type I error is a false alarm (a smoke alarm with no fire), while a Type II error is a missed detection (a fire with no alarm). This trade-off is crucial in A/B testing and medical diagnostics.
A/B Testing: Making Decisions with Data, Not Guesses
A/B testing is a controlled experiment pitting two versions of a product against each other with real users. It's used to see if a change, like a new button color, improves a metric like clicks.
Hypothesis Testing: Is Your Data Signal or Noise?
Hypothesis testing is a courtroom trial for your data: you assume a default 'null hypothesis' is true until your data provides enough evidence to reject it. It's used in A/B tests to validate changes.
Central Limit Theorem: Why Averages Form a Bell Curve
The Central Limit Theorem explains why averages of samples tend to form a bell curve, even if the original data doesn't. It's the foundation for A/B testing and quality control. The footgun is assuming it works for small or non-independent samples.
Outlier Detection: Finding Data That Doesn't Belong
Outlier detection finds data points that don't fit the pattern, signaling an error, fraud, or a new event. It's used to spot faulty sensor readings or fraudulent transactions.
Cluster Analysis: Finding Hidden Groups in Your Data
Cluster analysis automatically finds natural groupings in unlabeled data, like sorting a mixed bag of Legos without a manual. It's used for customer segmentation or anomaly detection.
Regression Analysis: Finding the Line of Best Fit
Regression analysis draws a line through data to model relationships between variables. Use it to predict a house price from its square footage or forecast sales from ad spend. The footgun: a strong correlation doesn't prove one variable causes the other.
Confidence Interval: Quantifying Uncertainty in Your Estimates
A confidence interval puts error bars around a measurement, showing the plausible range for a true value. It's used in A/B tests to report not just a winner, but the range of its likely impact.
P-value: Probability of Your Data, Not Your Hypothesis
A p-value measures how surprising your data is, assuming your null hypothesis (e.g., "no change") is true. It's used in A/B tests to decide if an effect is real. The footgun: a low p-value doesn't prove your theory, it just casts doubt on the null.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles