What is the multiple comparisons problem and how to correct?

WHAT THIS TESTS: This question probes your statistical maturity in experimentation. Interviewers want to know if you recognize that running many tests on the same data inflates the probability of at least one Type I error. It separates candidates who treat p-values as independent lottery tickets from those who understand the compounding risk of false positives across a family of tests.

A GOOD ANSWER COVERS: First, define the multiple comparisons problem in plain language. Each test has its own alpha, so the family-wise error rate grows with the number of tests. For example, with twenty independent metrics at alpha equals 0.05, the probability of at least one false positive is roughly 1 minus 0.95 to the twentieth power, or about 64 percent. Second, distinguish between the per-comparison error rate and the family-wise error rate. Third, explain practical corrections. The Bonferroni adjustment divides alpha by the number of tests, controlling the family-wise error rate but at the cost of statistical power. The Benjamini-Hochberg procedure controls the false discovery rate, which is less strict and often preferred when you are screening many metrics and can tolerate some false positives. Fourth, discuss the two scenarios in the prompt separately. Testing one control against five variants is a family of five comparisons, so you should adjust across those five. Looking at twenty metrics on a single A versus B test is a family of twenty metrics, so you should adjust across those twenty. Mention that the correction depends on how you define the family.

COMMON WRONG ANSWERS: A dangerous answer claims you only need to correct if you are running many experiments, not if you are looking at many metrics within one experiment. Another red flag is saying you will just look at the results and only report the significant ones, which is exactly the behavior that creates the problem. Some candidates suggest lowering the alpha threshold arbitrarily without naming a principled method. Others think splitting metrics into separate dashboards or analyses avoids the issue, which is false because the data and the peeking are still the same.

LIKELY FOLLOW-UPS: The interviewer might ask how Bonferroni compares to Benjamini-Hochberg in terms of power and when you would choose one over the other. They might ask what you would do if you have five variants and twenty metrics simultaneously, which requires thinking about hierarchical or composite corrections. They could also ask about the practical impact on sample size and experiment runtime, since stricter alpha thresholds demand larger n. Another follow-up is how Bayesian methods or sequential testing interact with multiplicity.

ONE CONCRETE EXAMPLE: Imagine you run an A/B test with one control and five variants, each compared to the control, and you track twenty metrics per variant. Without correction, you are conducting one hundred tests. At a 0.05 per-comparison alpha, you expect about five false positives purely by chance. A Bonferroni approach would require each test to pass at p less than 0.0005, which is severely underpowered for most product experiments. A more practical route is to control the false discovery rate across the twenty metrics per variant using Benjamini-Hochberg, or to pre-specify one primary metric and one variant of interest, reserving the rest for exploratory analysis with explicit disclaimers.

Source: Wikipedia: Multiple comparisons problem