Checking if a 5% A/B lift is significant
Hypothesis-testing fundamentals.
State hypotheses, check predetermined sample and power, run the right test for the metric, read p-value and confidence interval.
What's really being asked
The interviewer wants to know if you understand that a percentage lift means nothing without statistical context. It tests hypothesis testing, sample size, variance, and the discipline not to over-claim.
The full answer
First, frame it: the null hypothesis is no difference between variants, the alternative is a real difference. Confirm the experiment reached the sample size set by a prior power analysis for the minimum detectable effect, and that it ran the full predetermined duration so you are not peeking. Choose the test matching the metric: a two-proportion z-test for a binary engagement rate, a t-test or Mann-Whitney for continuous values, accounting for variance. Compute the p-value and, more usefully, the confidence interval on the lift. If the p-value is below the agreed alpha, commonly five percent, and the interval excludes zero, the result is statistically significant. Then assess practical significance, whether five percent matters for the business, and review guardrail metrics and sample ratio mismatch before trusting it.
The mistakes people make
Saying five percent lift is automatically significant. Ignoring sample size and variance entirely. Using a t-test on a skewed or binary metric. Reading only the p-value and never the confidence interval. Stopping the moment the dashboard looked good, then claiming significance.
What usually comes next
What sample size would you have needed beforehand. What is the difference between statistical and practical significance. How does a wide confidence interval change your recommendation.
A concrete example
Engagement is a binary 'engaged today' rate, so you run a two-proportion z-test on roughly fifty thousand users per arm. The five percent relative lift yields a p-value of 0.02 and a ninety-five percent confidence interval of one to nine percent, excluding zero. That is statistically significant at alpha 0.05. You then confirm SRM is clean and that a five percent engagement gain clears the bar product set for shipping, distinguishing significance from real-world value.
Interview question
A new variant shows a 5% lift. What single fact most undermines calling it statistically significant?
- a.The lift was measured in percentages rather than absolute counts
- b.Engagement is a binary metric instead of continuous
- c.The feature was tested on the new variant rather than the control
- d.The test had a tiny sample size with high variance and a confidence interval spanning zeroCorrect
Why? this is the answer
A small sample with high variance produces a confidence interval that includes zero, meaning the lift could be noise; the metric's units or type do not by themselves invalidate significance.
Just read this? Test yourself on what you have been reading.
- #ab-testing
- #statistics
- #hypothesis-testing
- #significance
- #confidence-interval
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on ab-testing — each one lists the topics its interview covers.
See open roles