Why is stopping an A/B test at first significance problematic?
Tests peeking and Type I error inflation. Name peeking; explain daily looks inflate false positive rates above nominal alpha; note p-values assume one look at fixed sample size; recommend pre-committed runtimes or sequential testing.
What's really being asked
Whether the candidate understands that frequentist statistical inference in A/B testing relies on a strict experimental protocol. Specifically, it tests if they know that repeatedly inspecting results and stopping at the first significant p-value invalidates the false positive rate guarantee. The interviewer wants to see familiarity with peeking, optional stopping, and why p-values are only valid under a pre-specified sample size and analysis plan.
The full answer
Four things in order. First, name the issue: this is the peeking problem, also called optional stopping. Second, explain the mechanism: every time you look at the data, you get another chance to observe a statistically significant result by random noise alone. Under a standard 5% significance level, a single analysis has a 5% false positive rate, but checking daily for 20 days can push the cumulative Type I error rate to roughly 20-30% or more depending on correlation across days. Third, clarify the statistical foundation: frequentist p-values and confidence intervals are computed assuming one look at the end of the experiment with a fixed sample size. Breaking that assumption means the reported p-value is no longer the true probability of seeing such an extreme result by chance. Fourth, offer a solution: pre-commit to a sample size and runtime before launch, do not stop early for significance, and if interim looks are truly necessary, use a sequential testing framework or alpha spending function to preserve the overall error rate.
The mistakes people make
Calling the issue generic p-hacking without distinguishing the specific repeated-testing mechanism of peeking. Saying the problem is simply that the sample size is too small, which misses the point that even a large sample can yield false positives if you repeatedly test it. Suggesting that switching to Bayesian methods automatically eliminates all peeking concerns, which is an oversimplification because Bayesian inference can still suffer from decision-theoretic peeking bias depending on priors and loss functions. Proposing to lower the alpha threshold arbitrarily without a formal multiple comparison correction.
What usually comes next
How would you design an experiment that needs interim results for business reasons? What is a sequential probability ratio test or group sequential design? How does peeking differ from the multiple testing problem across many metrics? When might Bayesian updating be a reasonable alternative?
A concrete example
Imagine a 30-day A/B test with a 5% significance level. If the product manager checks the dashboard every morning and plans to ship the winning variant the moment the p-value drops below 0.05, the actual probability of declaring a winner when there is no true effect is no longer 5%. Simulations show that with daily independent looks, the false positive rate can exceed 25%. If the manager instead pre-commits to running the full 30 days and only evaluates once at the end, the false positive rate stays at the intended 5%.
Interview question
In a 30-day A/B test checked daily for significance, why does the actual false positive rate exceed the nominal 5% level?
- a.Because Bayesian methods automatically eliminate concerns about optional stopping and peeking
- b.Because the alpha threshold can be arbitrarily lowered to offset the number of daily checks
- c.Because each daily look provides another chance to observe significance by random noise aloneCorrect
- d.Because the sample size is too small to reliably detect the true treatment effect
Why? this is the answer
The correct answer captures the peeking mechanism: every interim look gives random noise an additional chance to produce a significant p-value, inflating the cumulative Type I error rate far above the nominal 5%. The most tempting distractor, A, misattributes the inflated false positive rate to insufficient sample size rather than to the violation of the fixed-sample assumption.
Just read this? Test yourself on what you have been reading.
Read the original → docs.growthbook.io
- #ab testing
- #peeking
- #statistics
- #type i error
- #experimentation
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles