Why repeatedly extending a test inflates false positives
understanding the peeking problem and inflated false positives.
repeatedly checking and extending until significance is p-hacking via optional stopping, which inflates the false-positive rate; fix with fixed sample sizes or sequential…
What's really being asked
This checks statistical rigor under real-world pressure. The PM's request sounds reasonable but encodes a classic error. The interviewer wants you to name the peeking problem, explain why it inflates false positives, and offer a principled alternative rather than caving.
The full answer
The practice is called peeking, or optional stopping, a form of p-hacking. Classical fixed-horizon hypothesis tests are valid only if the sample size is fixed before the test and you evaluate significance once, at the end. The p-value fluctuates randomly over time, so if you repeatedly check and agree to stop as soon as it dips below the threshold, you are taking many chances to cross the line by luck. Each look adds opportunities for a false positive, so the true false-positive rate climbs far above the nominal five percent. Extending one more week because you are short of significance is exactly this: a data-dependent stopping rule that biases toward declaring a winner that does not exist. The right responses: run a power analysis up front to compute the required sample size and duration, then decide once when that is reached. If you genuinely need to monitor continuously, use methods built for it, such as sequential testing or always-valid p-values and confidence sequences, which adjust the thresholds to preserve the error rate. Also consider whether the effect is simply too small to matter.
The mistakes people make
Saying more data is always better, so extending is fine. Treating each weekly check as independent and harmless. Stopping the moment significance appears without any correction. Confusing low power, a real issue, with a license to peek. Ignoring the practical significance of the effect size.
What usually comes next
What is the fix if you must look daily? Sequential tests or alpha-spending. How do you set the sample size in advance? Power analysis from baseline rate, minimum detectable effect, and desired power. Why does the p-value wandering matter? It crosses thresholds by chance over many looks.
A concrete example
A team planned for two weeks but at one week sees p just above the threshold and extends, then extends again, stopping the instant p dips below it. Because they took repeated bites, the result that looks significant is likely noise. A colleague reruns the analysis with an always-valid sequential method and the effect is no longer significant, revealing the original call as a peeking-induced false positive.
Interview question
Why does repeatedly checking an A/B test and extending it until it reaches significance inflate the false-positive rate?
- a.More data makes the effect size artificially larger
- b.Longer tests always have less statistical power
- c.Extending the test changes the baseline conversion rate
- d.Each additional look is another chance for the fluctuating p-value to cross the threshold by luckCorrect
Why? this is the answer
A data-dependent stopping rule gives the noisy p-value many opportunities to dip below the threshold by chance, so the real error rate exceeds the nominal level. Extending does not reduce power, alter the baseline, or inflate the true effect size.
Just read this? Test yourself on what you have been reading.
Read the original → gopractice.io
- #peeking
- #p-hacking
- #optional-stopping
- #statistical-significance
- #sequential-testing
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles