Statistics
196 bites tagged Statistics — interview questions with model answers, and 60-second explainers.
How would you design drift detection for high-dimensional embeddings?
Tests distribution shift in latent spaces beyond per-feature stats. Strong answers use maximum mean discrepancy, k-NN two-sample tests, or domain-classifier AUC, plus windowing. Red flag: per-dimension KS tests or mean-difference thresholds.
Explain the difference between statistical and practical significance
Define statistical vs practical significance; note large samples make tiny effects significant; give a real example. If you know p-values detect real effects but ignore value. Using p < 0.05 alone to justify shipping.
How do you prevent concurrent onboarding and navigation experiments from polluting results?
This tests experiment isolation via layer-based traffic allocation. A strong answer covers hashing users into independent layers with one variant per layer, and assigning each experiment to a distinct layer.
What is the multiple comparisons problem and how to correct?
This tests your grasp of family-wise error inflation across many tests. A strong answer defines the problem, contrasts per-comparison and family-wise error, and names corrections like Bonferroni or FDR.
Explain statistical power, MDE, and sample size impact
Power is the chance of detecting a true effect; MDE is the smallest lift worth measuring; raising power or shrinking MDE inflates sample size. This checks whether you grasp the power-MDE-sample-size trade-off.
What is a p-value in A/B tests and what does threshold mean?
P-value is the chance of this or more extreme result if the null is true; the 5% threshold is the accepted false-positive rate. your grasp of frequentist testing mechanics. saying it is the probability the variant wins.
How do you set up a button color A/B test?
Tests whether you can define a valid experiment, not just a color change. A strong answer covers hypothesis, randomization, primary metric, sample size, and significance threshold.
WAU is flat despite positive A/B tests; why and how to diagnose
This tests distinguishing real impact from statistical artifacts. Strong answers cite false positives from low base rates, peeking, novelty, and local-global mismatches. Diagnose with long-term holdouts, audits, and causal bridges.
Why not stop an A/B test when it looks significant early?
Tests whether you understand repeated looks inflate false positives. The term is peeking: checking daily can turn a 5% Type I error rate into roughly 15% by day 3. Red flag: citing "low sample size" without stating that early stopping invalidates the p-value.
A/B test: 0.1% lift. Statistical vs practical significance?
Statistical significance says the 0.1% is real; practical significance asks if revenue exceeds engineering cost. Frame with CIs and ROI. Distinguishing proof from business value. Shipping just because p is low while ignoring cost.
P-value vs confidence interval in an A/B test
A p-value gauges evidence against the null; a 95% CI shows plausible effect sizes and precision. Frequentist reasoning beyond binary significance. Calling the CI a 95% probability the true difference is inside.
How do you determine sample size for a conversion lift experiment?
Tests fluency with statistical experiment design. A strong answer frames N as a function of alpha, power, baseline rate, and MDE, noting that shrinking the MDE or raising power inflates N. Red flag: picking N from traffic instead of risk tolerance.
Explain the Central Limit Theorem and its importance for hypothesis testing
This tests whether you know why sample means from skewed populations tend toward normal as size grows, enabling tests. A strong answer covers mean convergence to normal and standard error. Red flag: claiming the CLT works for small samples or single points.
What is a p-value? Interpret p = 0.03 at alpha = 0.05.
Tests frequentist testing and p-value misinterpretations. Define p-value as the probability of data this extreme under the null; since 0.03 < 0.05, reject the null at 5%. Never say it is the probability the null is false or the result is due to chance.
What are your null and alternative hypotheses for this A/B test?
This tests translating a directional business question into statistical hypotheses. A strong answer states H0 as no difference in registration rate and H1 as green outperforming blue. A red flag is framing H0 as "blue is better" or using a two-tailed test.
How would you recommend launching a checkout flow with mixed A/B metrics?
This tests multi-metric trade-offs. A strong answer tags conversion as success and AOV as a guardrail, estimates net revenue impact, and frames decision as a risk-managed business choice. A red flag is demanding all metrics win or ignoring business context.
MLE: Find the Parameters That Make Data Likely
MLE tunes your model until observed data looks inevitable. Use it to fit distributions to logs, traffic, or errors. The footgun: it assumes your distribution family is correct; under a wrong model, it finds the best-fitting wrong answer with high confidence.
How do you diagnose a confounded A/B test with slower page load?
This tests confounding beyond p-values. A strong answer isolates the bug, checks if slower load hurt or helped conversions, and judges whether copy or latency caused the lift. A red flag is defending a 5% win because p < 0.05 while ignoring randomization.
Compare A/B/n testing with multi-armed bandits for headline optimization
Contrast A/B/n with dynamic MAB allocation; A/B/n optimizes final inference, MABs optimize reward; flag real-time infra. Grasp of explore-exploit and when rigor beats speed. Calling MABs always better ignoring regret.
Explain statistical significance in copy A/B tests and why one day fails.
This checks if you distinguish signal from noise. A strong answer defines statistical significance as confidence a difference is real, warns that one-day samples are small and skewed by variance, and cites false positive risk.
What statistical methods automate canary-baseline comparison and handle noise?
Tests statistical rigor in automated canary analysis. Strong answers use non-parametric tests, multi-metric aggregation with effect-size gates, MAD-based outlier rejection, and smoothing windows.
What is the difference between a primary metric and a guardrail metric?
Tests whether you distinguish success criteria from safety checks in experiments. A strong answer defines primary metrics as the target outcome, guardrails as protective thresholds, and gives a concrete scenario where a primary lift does not justify shipping…
Why is stopping an A/B test at first significance problematic?
Tests peeking and Type I error inflation. Name peeking; explain daily looks inflate false positive rates above nominal alpha; note p-values assume one look at fixed sample size; recommend pre-committed runtimes or sequential testing.
How do you determine sample size and duration for an A/B test?
This tests statistical power literacy. A strong answer names baseline rate, MDE, alpha, and beta; explains the duration versus sensitivity trade-off; and notes traffic allocation. A red flag is ignoring power or stopping early when results look significant.
Get Statistics bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.