Checking if a 5% A/B lift is significant

WHAT THIS TESTS The interviewer wants to know if you understand that a percentage lift means nothing without statistical context. It tests hypothesis testing, sample size, variance, and the discipline not to over-claim.

A GOOD ANSWER COVERS First, frame it: the null hypothesis is no difference between variants, the alternative is a real difference. Confirm the experiment reached the sample size set by a prior power analysis for the minimum detectable effect, and that it ran the full predetermined duration so you are not peeking. Choose the test matching the metric: a two-proportion z-test for a binary engagement rate, a t-test or Mann-Whitney for continuous values, accounting for variance. Compute the p-value and, more usefully, the confidence interval on the lift. If the p-value is below the agreed alpha, commonly five percent, and the interval excludes zero, the result is statistically significant. Then assess practical significance, whether five percent matters for the business, and review guardrail metrics and sample ratio mismatch before trusting it.

COMMON WRONG ANSWERS Saying five percent lift is automatically significant. Ignoring sample size and variance entirely. Using a t-test on a skewed or binary metric. Reading only the p-value and never the confidence interval. Stopping the moment the dashboard looked good, then claiming significance.

LIKELY FOLLOW-UPS What sample size would you have needed beforehand. What is the difference between statistical and practical significance. How does a wide confidence interval change your recommendation.

ONE CONCRETE EXAMPLE Engagement is a binary 'engaged today' rate, so you run a two-proportion z-test on roughly fifty thousand users per arm. The five percent relative lift yields a p-value of 0.02 and a ninety-five percent confidence interval of one to nine percent, excluding zero. That is statistically significant at alpha 0.05. You then confirm SRM is clean and that a five percent engagement gain clears the bar product set for shipping, distinguishing significance from real-world value.