Why not stop an A/B test when it looks significant early?

WHAT THIS TESTS: This question probes whether you understand that frequentist statistical inference depends on a fixed experimental protocol. In a standard t-test or z-test, the p-value is mathematically derived under the assumption that the sample size and stopping rule are determined before the experiment begins. If you violate that protocol by checking repeatedly and stopping conditionally, you are no longer playing the game the p-value was designed for.

A GOOD ANSWER COVERS: First, name the problem: it is called peeking, also known as optional stopping or interim monitoring without correction. Second, explain the mechanism: every time you look at the data, you give the random noise another chance to cross the significance threshold. With a nominal 5% alpha and daily checks, the actual false positive rate can exceed 15% within just a few days because the probabilities accumulate. Third, distinguish it from ordinary sample size concerns: even if the effect is real, early estimates are volatile and often regress to the mean as more users enter the experiment. Fourth, offer a solution: pre-commit to a runtime and sample size, use a proper power analysis, or adopt a sequential testing framework such as group sequential tests or always-valid p-values if the business truly needs early stopping.

COMMON WRONG ANSWERS: Saying "the sample is too small" without addressing why the p-value itself is invalidated. Calling it the multiple testing problem; while related, that term usually refers to testing many metrics or subgroups, not repeated temporal looks at the same metric. Suggesting to lower the alpha threshold ad hoc without a formal correction method like O'Brien-Fleming or spending functions. Agreeing to stop early because "the PM needs an answer fast," which signals you will trade statistical rigor for speed.

LIKELY FOLLOW-UPS: How would you design a test if leadership insists on daily readouts? What is the difference between peeking under frequentist versus Bayesian frameworks? Can you explain why a 5% daily alpha does not equal a 5% experiment-wise alpha? How do sequential testing or always-valid p-values solve this mathematically?

ONE CONCRETE EXAMPLE: Imagine you run a 30-day experiment with a 5% significance level and you peek every day. The probability of at least one false positive across those 30 looks is roughly 1 minus 0.95 to the 30th power, which is about 79%. Even though each individual day carries a 5% risk, the compound risk is catastrophic. In practice, with correlated daily observations the exact inflation is slightly less, but the directional problem remains severe: by day 3 you might easily have a 15% to 20% chance of declaring a winner when there is no true effect.

Source: GrowthBook Docs