The multiple comparisons problem in A/B testing

WHAT THIS TESTS Whether you grasp that statistical significance assumes a single planned comparison, and that running many comparisons silently inflates false positives, plus how to control it both statistically and through platform design.

A GOOD ANSWER COVERS Each hypothesis test at alpha 0.05 accepts a 5 percent chance of a false positive when there is truly no effect. Run twenty independent such tests under the null and the probability of at least one false positive rises toward 64 percent; this is the family-wise error rate, and it grows with the number of tests and the number of metrics examined per test. At a platform running dozens of concurrent experiments, naive per-test thresholds guarantee spurious winners, leading teams to ship changes that do nothing. Mitigation has two arms. Statistically, the Bonferroni correction divides alpha by the number of comparisons; it is simple but very conservative and costs power. The Benjamini-Hochberg procedure instead controls the false discovery rate, the expected proportion of false positives among the wins, which is far better suited to running many experiments because it preserves more power.

SYSTEM DESIGN Pre-register one primary success metric per experiment, treat the rest as secondary or guardrail metrics, enforce minimum sample size and runtime, use sequential testing methods that allow valid peeking, and ensure traffic-splitting isolation so concurrent tests do not interfere.

COMMON WRONG ANSWERS Treating each test as fully independent and ignoring the family of comparisons. Scanning every metric and declaring victory on whichever hits p<0.05, which is metric fishing. Peeking repeatedly and stopping at first significance. Always using Bonferroni even when it destroys power.

ONE CONCRETE EXAMPLE A team tracks fifteen metrics on one experiment; three show p<0.05 by chance. Applying Benjamini-Hochberg, only the metric with a genuinely tiny p-value survives the FDR threshold, so the team correctly concludes the variant moved one real metric rather than shipping three illusory wins it would have claimed under naive testing.