Statistics
196 bites tagged Statistics — interview questions with model answers, and 60-second explainers.
Randomized Controlled Trials: The Gold Standard for 'Did It Work?'
An RCT is a science fair experiment for business decisions, isolating one change to see its true effect. It's used in A/B tests to prove a new feature worked. The biggest footgun is peeking at results early, which can lead to false conclusions.
MaxDiff Analysis: Find True Preferences, Not Just Ratings
MaxDiff finds what people truly value by asking them to pick the "best" and "worst" from a small set, not just rate them. Use it to rank features or messages without the ambiguity of 1-5 scales.
P-Value: Gauging Surprise, Not Certainty
A p-value measures surprise: it's the probability of seeing your results by chance, assuming your change had no effect. It's used in A/B testing to decide if an effect is noise or significant. A small p-value doesn't prove your hypothesis is true.
Snowball Sampling: When Your Users Find Your Users
Snowball sampling has your first participants recruit the next ones from their network. It's vital for reaching hidden groups, like specific professional communities. The footgun is selection bias: you're sampling social networks, not the whole population.
Longitudinal Study: Watching Change Over Time
Think of it as a movie, not a snapshot. A longitudinal study tracks the same users over time to see how behavior evolves, revealing cause and effect. It's used to measure long-term feature adoption or churn.
UX Sampling: Convenience vs. Probability
Convenience sampling is fast and cheap—ask whoever is easy to reach. Probability sampling is rigorous—ask a random slice of your population. Use convenience for quick usability tests, but probability for high-stakes decisions.
Time-Series Forecasting: Predicting the Future from the Past
Time-series forecasting uses past data points, ordered by time, to predict future values. It's used for capacity planning and financial modeling. The footgun is assuming past trends will hold, as sudden system changes can invalidate all predictions.
Population Stability Index (PSI): Quantifying Data Drift
The Population Stability Index (PSI) gives you a single number to quantify data drift between training and live data. It's used in MLOps to monitor model health, especially in finance. The footgun is ignoring a high PSI, which signals silent prediction decay.
Regularization: Penalizing Complexity to Prevent Overfitting
Regularization penalizes model complexity to prevent overfitting. It's used in training to help models generalize to new data, rather than just memorizing training examples. The footgun is applying too much, causing the model to become too simple and underfit.
What is an Experimentation Stats Engine?
A stats engine is the brain of an A/B testing platform, turning raw data into reliable 'which version won?' decisions. It powers tools that analyze feature rollouts, ensuring statistical rigor.
Causal Impact: Measuring Effects Without an A/B Test
Causal Impact estimates an intervention's effect by modeling a 'what if' counterfactual. It's used to measure lift from a new feature or ad campaign when a clean A/B test isn't possible.
Forecasting with Monte Carlo Simulation
A Monte Carlo simulation forecasts outcomes by running thousands of 'what if' scenarios with random inputs. Use it to model complex systems like user growth with variable conversion rates.
Survival Analysis: Predicting When, Not Just If
Survival analysis predicts *when* an event like user churn will happen, not just *if*. It's used to model customer lifetime or hardware failure rates. The key mistake is using simple averages, which are skewed by users who haven't churned yet (censored data).
Prophet: Time Series Forecasting for Seasonal Data
Prophet treats a time series as a sum of its parts: a long-term trend, seasonal cycles, and holidays. It's used for business forecasting, like predicting sales, when you have strong seasonal data. The footgun is using it for non-seasonal data.
Linear Regression: Forecasting with a Straight Line
Linear regression forecasts the future by drawing a straight line through past data. It's used to predict outcomes like sales based on ad spend or energy use based on temperature.
Conjoint Analysis: What Features Do Users *Really* Value?
Stop asking users what they want; make them choose. Conjoint analysis reveals true priorities by forcing trade-offs between product features, like price vs. battery life. It's used for pricing and roadmapping.
Bonferroni Correction: Raising the Bar for Significance
The Bonferroni correction prevents finding false positives when running many tests by making your significance threshold stricter for each one. It's used in A/B tests with multiple variants.
Regression to the Mean: Why Outliers Settle Down
Extreme results are part skill, part luck. Regression to the mean is the principle that luck evens out, so a follow-up measurement will be closer to the average. This impacts A/B tests and performance analysis.
Chi-Squared Test: Are These Two Things Related?
A Chi-Squared test detects 'surprising' differences between what you observe and what you'd expect. It's used to check if two categorical variables, like a landing page variant and a user's sign-up action, are independent or related.
Student's t-test: Is This Difference Real or Just Noise?
A t-test tells you if the difference between two group averages is statistically significant, especially with small samples. It's used in A/B testing to see if a new feature actually improved a metric, or if the change is just noise.
Randomization: The Key to Trustworthy Experiments
Randomization ensures experimental groups are similar before a test begins, like shuffling cards before dealing. This lets you confidently attribute differences in outcomes to your changes, whether in A/B tests or clinical trials.
Null Hypothesis: Assume No Effect Until Proven Otherwise
The null hypothesis is your default assumption: nothing changed. You run experiments to gather enough evidence to reject this default. In A/B testing, the null is that your new feature has no effect, while the alternative is that it does.
Minimum Detectable Effect: How Small a Change Can You See?
Minimum Detectable Effect (MDE) is the smallest change your A/B test can reliably see. You calculate it *before* a test to determine the sample size needed.
Log Transformation: Taming Skewed Data for Better Models
A log transform tames skewed data by compressing large values and spreading out small ones. It's used on data like income or web traffic to help it meet the assumptions of linear models. The footgun: it fails on zero or negative values.
Get Statistics bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.