Statistics
196 bites tagged Statistics — interview questions with model answers, and 60-second explainers.
Why can't you t-test p99 latency, and what's a valid alternative?
Explain that t-tests target means while p99 variance depends on tail density; propose bootstrap CIs or permutation tests. Knowing percentiles are order statistics, not means. Invoking CLT to justify a t-test on p99.
Explain Simpson's Paradox and construct a user engagement scenario
Tests whether you spot trends reversing when population mixes differ. Good answers define the paradox, give a numerical example with per-segment wins but aggregate loss, and warn against segment-only decisions.
Determine if a 10% DAU drop is statistically significant
Tests signal vs noise in stable metrics. Good answers define a null hypothesis, compute a test statistic from historical variance, compare to a critical value at set alpha, and check seasonality. Red flag: calling a large drop real without baseline variance.
A/B test p-value 0.08, PM wants to ship. How do you advise?
Tests statistical rigor versus business pragmatism. A strong answer covers pre-registered thresholds, false positive risk, statistical power, confidence intervals, and the business cost of being wrong. Red flag: shipping without quantifying downside risk.
What does a p-value of 0.03 mean at alpha 0.05?
0.03 means 3% chance of data this extreme if the null (no effect) holds; since 0.03 < 0.05, reject the null. Literal p-value interpretation. Calling it the probability the null is true or the effect is 97% real.
How would you visually represent statistical uncertainty in a chart?
Awareness that plotted points are perceived as exact truths. Replace isolated bars with intervals showing point estimate uncertainty; add hypothetical outcome plots to make values tangible. Offering p-values or raw means without visualizing uncertainty range.
Why prefer median and p95 over mean for API latency?
This tests statistical intuition for skewed distributions. A strong answer notes that median captures typical experience, p95 captures tail suffering, and mean hides outliers. A red flag is claiming mean alone is sufficient.
Determine Sample Size for a 2% Lift A/B Test
This tests your grasp of statistical power and the business trade-offs in experimentation. A great answer defines baseline conversion rate, minimum detectable effect (MDE), and statistical power. A red flag is ignoring the business context of MDE.
What is a p-value, and what does 0.03 practically mean?
This tests your ability to translate stats into business decisions. A great answer defines p-value, compares 0.03 to the standard 0.05 threshold to reject the null hypothesis, and recommends shipping.
How do you handle the multiple comparisons problem in A/B testing?
Tests your grasp of statistical risk in experimentation. Explain how multiple tests inflate false positives, then describe mitigations like Bonferroni correction or limiting concurrent tests. A red flag is suggesting total test isolation, which is impractical.
Design an A/B Testing Framework
This tests your ability to design a scalable system with statistical rigor for non-experts. A great answer outlines config management, deterministic user hashing, a data pipeline for metrics, and a results UI that simplifies stats.
Design an experiment to isolate long-term impact from novelty effect
Tests if you can design experiments for long-term impact, not just short-term lift. A good answer involves a long-running test, segmenting users by tenure, and modeling the effect over time to find its stable asymptote.
Describe two methods for generating prediction intervals
This tests your grasp of uncertainty quantification. A great answer contrasts an analytical method (assuming normal errors, using multipliers like 1.96 for 95%) with a simulation method (bootstrapping residuals).
Which model for forecasting with seasonality and trend?
This tests your knowledge of classical time series models. A good answer names Holt-Winters, explaining its level, trend, and seasonal components. It also discusses choosing between additive and multiplicative methods. A red flag is jumping to complex models.
Explain time series stationarity and how to achieve it
Tests your grasp of core time series assumptions. Define stationarity (constant mean/variance over time), explain why models need it for stable predictions, and name methods to test and achieve it. A red flag is just saying the data looks 'flat'.
How CUPED increases statistical power in experiments
Tests your grasp of variance reduction in A/B testing. Explain how CUPED uses correlated pre-experiment data to reduce outcome variance, increasing statistical power. A red flag is confusing it with simpler difference scores, which can actually increase noise.
Why is stopping an A/B test early problematic?
Tests understanding of the 'peeking problem' in A/B testing. A good answer defines peeking, explains how it inflates false positive rates, and contrasts it with waiting for a pre-determined sample size. A red flag is not explaining the statistical mechanism.
How do you determine sample size and duration for an A/B test?
This tests your grasp of statistical power and business trade-offs. A good answer defines the four inputs (baseline, MDE, significance, power) to calculate sample size, then uses traffic to find duration.
Why can't we t-test p99 latency in an A/B test?
This tests if you know why t-tests fail for percentiles. A t-test requires a normally distributed statistic (like the mean), but a sample p99's distribution isn't normal.
Explain Simpson's Paradox with a user engagement example
Tests if you see beyond aggregate data. Define the paradox, give a numerical example where a feature fails overall but wins in segments (e.g., new vs. returning users), and name the confounding variable. A vague definition without numbers is a red flag.
DAU dropped 10% overnight. Is this a significant change?
Tests your use of statistical hypothesis testing on business metrics. Outline the process: state a null hypothesis (no change), choose a Z-test, calculate the p-value, and compare to an alpha of 0.05. A red flag is guessing causes before proving significance.
A/B test p-value is 0.08, PM wants to ship. What now?
Tests if you can translate statistical risk into business terms for a PM. A good answer defines the 8% false positive risk, weighs it against the cost of shipping, and suggests next steps like running the test longer instead of just saying no.
What does a p-value of 0.03 mean in an A/B test?
This tests your grasp of statistical significance and ability to make a data-driven decision. A great answer defines p-value, compares it to alpha (0.03 < 0.05), and concludes to reject the null hypothesis. A red flag is misstating the p-value's definition.
How would you validate a feature's success, correcting for self-selection?
This tests your ability to separate correlation from causation. A great answer identifies self-selection bias, proposes a quasi-experimental method like Propensity Score Matching to create a control group, and then compares outcomes.
Get Statistics bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.