What is the 'multiple comparisons problem' in A/B testing?
Tests your grasp of statistical pitfalls in large-scale A/B testing. Define the problem (inflated false positives), explain the business risk (wasted effort), and propose a mitigation like Bonferroni correction.
What's really being asked
This question tests your understanding of how statistical significance erodes at scale. Interviewers want to see if you can connect a theoretical statistics concept to a real-world engineering problem—running a large A/B testing platform—and propose practical, not just academic, solutions. They are assessing your ability to balance development velocity with statistical validity.
The full answer
A strong answer addresses four key points in order. First, define the problem: as the number of concurrent A/B tests increases, the probability of observing at least one false positive (a 'significant' result that is just random noise) grows dramatically. Second, quantify the risk: with a standard p-value of 0.05, running 20 tests increases the chance of at least one false positive to over 64% (calculated as 1 - 0.95^20). Third, explain the business impact: this leads to wasted engineering effort building and shipping features that have no real impact, which in turn erodes trust in the experimentation platform. Fourth, describe mitigations: discuss at least one statistical method and one system design approach. For statistics, contrast the conservative Bonferroni correction (dividing the p-value threshold by the number of tests) with the more balanced Benjamini-Hochberg procedure (which controls the false discovery rate). For system design, you could discuss managing test interference, while also noting that research from firms like Microsoft suggests harmful interactions are rarer than often feared.
The mistakes people make
A major red flag is dismissing the problem as purely academic or irrelevant. Another is overcorrecting by suggesting a halt to concurrent testing or running only one test at a time; this is impractical for most platforms and shows a lack of business awareness. A weaker answer might identify the problem but fail to quantify the risk or offer concrete, named solutions. Simply saying 'we should be careful' is insufficient for a senior role.
What usually comes next
Expect follow-ups like: 'How would you choose between Bonferroni and Benjamini-Hochberg?' (Answer: It's a trade-off between your tolerance for false positives versus missing real effects). Or, 'What about test interference, where one test influences another?' (Answer: It's a related but distinct problem, often managed with user segmentation or by accepting that most interactions are minor). A product-focused follow-up might be: 'How would you explain this to a PM who just wants to ship their 'winning' feature?'
A concrete example
If your platform runs 20 concurrent tests, each aiming for a 95% confidence level (p=0.05), you have a 64% chance of at least one test showing a false positive. If a 'winning' test from this batch showed a 5% lift, but a Bonferroni correction was applied, the new required p-value would be 0.05 / 20 = 0.0025. The original result would likely no longer be statistically significant, correctly preventing the team from shipping a feature based on a statistical fluke.
Interview question
When running multiple A/B tests concurrently, what is the primary statistical challenge posed by the "multiple comparisons problem"?
- a.A reduction in the statistical power of each individual test, making it harder to detect true effects.
- b.The need for larger sample sizes for each test to maintain the desired confidence level.
- c.A significant increase in the probability of observing at least one false positive result across all tests.Correct
- d.An increased risk of test interference, where one experiment's effect contaminates another's results.
Why? this is the answer
The multiple comparisons problem primarily leads to a dramatically increased probability of false positives (Type I errors) when many tests are conducted simultaneously. While test interference (option D) is a related concern, the card explicitly states it is a distinct problem from multiple comparisons.
Just read this? Test yourself on what you have been reading.
Read the original → statsig.com
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on a/b testing — each one lists the topics its interview covers.
See open roles