Why repeatedly extending a test inflates false positives

WHAT THIS TESTS This checks statistical rigor under real-world pressure. The PM's request sounds reasonable but encodes a classic error. The interviewer wants you to name the peeking problem, explain why it inflates false positives, and offer a principled alternative rather than caving.

A GOOD ANSWER COVERS The practice is called peeking, or optional stopping, a form of p-hacking. Classical fixed-horizon hypothesis tests are valid only if the sample size is fixed before the test and you evaluate significance once, at the end. The p-value fluctuates randomly over time, so if you repeatedly check and agree to stop as soon as it dips below the threshold, you are taking many chances to cross the line by luck. Each look adds opportunities for a false positive, so the true false-positive rate climbs far above the nominal five percent. Extending one more week because you are short of significance is exactly this: a data-dependent stopping rule that biases toward declaring a winner that does not exist. The right responses: run a power analysis up front to compute the required sample size and duration, then decide once when that is reached. If you genuinely need to monitor continuously, use methods built for it, such as sequential testing or always-valid p-values and confidence sequences, which adjust the thresholds to preserve the error rate. Also consider whether the effect is simply too small to matter.

COMMON WRONG ANSWERS Saying more data is always better, so extending is fine. Treating each weekly check as independent and harmless. Stopping the moment significance appears without any correction. Confusing low power, a real issue, with a license to peek. Ignoring the practical significance of the effect size.

LIKELY FOLLOW-UPS What is the fix if you must look daily? Sequential tests or alpha-spending. How do you set the sample size in advance? Power analysis from baseline rate, minimum detectable effect, and desired power. Why does the p-value wandering matter? It crosses thresholds by chance over many looks.

ONE CONCRETE EXAMPLE A team planned for two weeks but at one week sees p just above the threshold and extends, then extends again, stopping the instant p dips below it. Because they took repeated bites, the result that looks significant is likely noise. A colleague reruns the analysis with an always-valid sequential method and the effect is no longer significant, revealing the original call as a peeking-induced false positive.