The multiple comparisons problem in A/B testing
Statistical rigor at scale.
Many tests at alpha 0.05 inflate the chance of a false positive; mitigate with Bonferroni or FDR control plus pre-registered metrics.
Cherry-picking whichever metric crosses p<0.05.
WHAT THIS TESTS Whether you grasp that statistical significance assumes a single planned comparison, and that running many comparisons silently inflates false positives, plus how to control it both statistically and through platform design.
A GOOD ANSWER COVERS Each hypothesis test at alpha 0.05 accepts a 5 percent chance of a false positive when there is truly no effect. Run twenty independent such tests under the null and the probability of at least one false positive rises toward 64 percent; this is the family-wise error rate, and it grows with the number of tests and the number of metrics examined per test. At a platform running dozens of concurrent experiments, naive per-test thresholds guarantee spurious winners, leading teams to ship changes that do nothing. Mitigation has two arms. Statistically, the Bonferroni correction divides alpha by the number of comparisons; it is simple but very conservative and costs power. The Benjamini-Hochberg procedure instead controls the false discovery rate, the expected proportion of false positives among the wins, which is far better suited to running many experiments because it preserves more power.
SYSTEM DESIGN Pre-register one primary success metric per experiment, treat the rest as secondary or guardrail metrics, enforce minimum sample size and runtime, use sequential testing methods that allow valid peeking, and ensure traffic-splitting isolation so concurrent tests do not interfere.
COMMON WRONG ANSWERS Treating each test as fully independent and ignoring the family of comparisons. Scanning every metric and declaring victory on whichever hits p<0.05, which is metric fishing. Peeking repeatedly and stopping at first significance. Always using Bonferroni even when it destroys power.
ONE CONCRETE EXAMPLE A team tracks fifteen metrics on one experiment; three show p<0.05 by chance. Applying Benjamini-Hochberg, only the metric with a genuinely tiny p-value survives the FDR threshold, so the team correctly concludes the variant moved one real metric rather than shipping three illusory wins it would have claimed under naive testing.
Interview question
At a platform running many concurrent experiments, why is Benjamini-Hochberg often preferred over Bonferroni for handling multiple comparisons?
- a.It removes the need to pre-register a primary metric
- b.It eliminates false positives entirely rather than just limiting them
- c.It controls the false discovery rate while preserving more statistical powerCorrect
- d.It allows using a higher alpha than 0.05 on every test
Why? this is the answer
Benjamini-Hochberg controls the expected proportion of false discoveries and is less conservative than Bonferroni, retaining power at scale. It does not eliminate all false positives nor replace good experiment design.
Just read this? Test yourself on what you have been reading.
Read the original → statsig.com
- #ab-testing
- #multiple-comparisons
- #statistics
- #false-discovery-rate
- #experimentation
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on ab-testing — each one lists the topics its interview covers.
See open roles