Peeking in A/B tests and how to mitigate it

WHAT THIS TESTS This evaluates statistical literacy around optional stopping: why monitoring a fixed-horizon test continuously and stopping at the first significant moment breaks the guarantees.

A GOOD ANSWER COVERS What peeking is: repeatedly checking a running experiment's p-value and stopping as soon as it crosses the significance threshold. Why it is bad: a fixed-sample significance test assumes one decision at a planned sample size; each additional look is another opportunity for random fluctuation to dip below 0.05, so across many peeks the chance of a false positive is far higher than the nominal 5 percent. The observed 5 percent lift may be a high-water mark caught mid-fluctuation. Mitigations: pre-compute the required sample size from a power analysis and only evaluate once that horizon is reached; or, if the team wants to watch continuously, adopt methods built for it, sequential testing with alpha spending or always-valid p-values and confidence sequences that hold under continuous monitoring. Engineering-wise, gate the decision on reaching the planned sample and surface dashboards that discourage early calls.

COMMON WRONG ANSWERS Claiming any look at a significant p-value justifies shipping. Treating the nominal 0.05 as valid despite dozens of peeks. Just running longer without a planned horizon or sequential correction. Confusing a larger sample with immunity to optional stopping.

LIKELY FOLLOW-UPS Why does each peek inflate the false positive rate? How does sequential testing or alpha spending fix it? How do you set the fixed sample size up front?

ONE CONCRETE EXAMPLE A team watches the pricing test hourly; on day two p hits 0.04 at a 5 percent lift and they want to ship. Because they peeked dozens of times, the real false-positive risk is well above 5 percent. The disciplined path: either wait for the pre-computed sample size and decide once, or switch to an always-valid sequential method whose threshold accounts for continuous looks.