How do you handle the multiple comparisons problem in A/B testing?

What's really being asked

Your understanding of a core statistical pitfall in modern, high-velocity A/B testing platforms. The interviewer is checking if you can connect statistical theory (inflated false positive rates) to real-world business impact (wasting resources on fake "wins") and propose practical, balanced solutions, not just textbook answers. They want to see you can balance speed with rigor.

The full answer

A good answer addresses three points in order. First, define the multiple comparisons problem: with each test having a chance of a false positive (e.g., 5% at p=0.05), running many tests concurrently dramatically increases the probability of getting at least one bogus "significant" result. For example, running 20 tests gives a 64% chance of at least one false positive. Second, explain the business cost: engineering and product resources are wasted implementing changes that have no real impact, while true opportunities are missed. Third, describe two categories of mitigation. For statistical methods, mention the conservative Bonferroni correction (dividing the p-value threshold by the number of tests) and the more balanced Benjamini-Hochberg procedure (which controls the false discovery rate). For system/process design, suggest limiting the number of concurrent tests or using a platform that automates these corrections.

The mistakes people make

A major red flag is ignoring the statistical explanation and focusing only on system performance or engineering complexity. Another is proposing to solve the problem by completely isolating all tests into separate user segments. This is often impractical, slows down velocity, and fails to test feature combinations that users will actually experience in production. Finally, candidates who seem unaware that this is a problem at all, or who suggest "just ship the winners," are demonstrating a lack of statistical maturity required for a senior role. Overcorrecting is also a mistake; suggesting methods so conservative that no test could ever reach significance shows a lack of practical balance.

What usually comes next

"How would you choose between Bonferroni and Benjamini-Hochberg?" (Bonferroni is for when you absolutely cannot tolerate a single false positive; B-H is better for discovery when you can accept a small, controlled rate of false discoveries). "What about test interference, where one test's effects alter the results of another?" (Acknowledge it's a concern, but mention that major platforms have found harmful interactions to be rare. The bigger risk is often the multiple comparisons problem itself). "How would you build a system to automate this?" (Describe a service that tracks active experiments, applies a chosen correction method to p-values before results are reported, and flags results accordingly).

A concrete example

Imagine we run 20 concurrent A/B tests, each with a standard p-value threshold of 0.05 for significance. This means each test has a 1-in-20 chance of being a false positive. Across all 20 tests, the probability of having at least one false positive isn't 5%, it's 1 - (0.95)^20, which is about 64%. To mitigate this with Bonferroni correction, we would adjust our significance threshold to 0.05 / 20 = 0.0025. A test result would now need a p-value below 0.0025 to be considered significant, drastically reducing the chance of a false positive making it through.