The Multiple Comparisons Problem

The mental model

Running many statistical tests on the same data is like giving yourself multiple chances to be fooled by randomness. If you run one test with a 95% confidence level (p=0.05), there's a 5% chance of a false positive. If you run 20 such tests, the probability of getting at least one false positive isn't 5%; it's much higher. You're giving random chance 20 opportunities to produce a fluke result that looks significant. This is the multiple comparisons problem, also called multiplicity.

How it works

The math shows how quickly the error rate grows. The probability of not making a Type I error in one test is (1 - α), where α is your significance level (e.g., 0.05). For 'k' independent tests, the probability of making no errors across all of them is (1 - α) to the power of k. Therefore, the probability of making at least one error, known as the family-wise error rate (FWER), is 1 - (1 - α)^k. For α=0.05 and k=20 tests, the FWER is 1 - (0.95)^20, which is approximately 64%. Your effective error rate has ballooned from 5% to 64%.

When to use it

This is a problem to be aware of, not a tool to use. It's critical in three main situations. First, in A/B testing when you track many metrics or segment users into many groups for the same experiment. Second, in scientific studies like genomics where thousands of genes are tested for association with a disease. Third, in any exploratory data analysis where you are "fishing" for correlations by testing many pairs of variables.

When not to use it

The problem is most acute when tests are performed to confirm a hypothesis. If you have a single, pre-specified hypothesis before looking at the data, you don't need to correct for other tests you could have run. The correction applies only to the set of tests you actually perform. It's also less of a concern in pure exploration, as long as any "discoveries" are treated as tentative and are later validated with a new, independent experiment designed to test that single finding.

One canonical example

An engineering team A/B tests a website redesign. They track 20 different metrics: button clicks, time on page, bounce rate, add to cart, newsletter sign-ups, etc., with a significance level of p < 0.05 for each. Even if the redesign has no real effect, there is a high probability (~64%) that at least one metric will show a "statistically significant" change just by random chance. The team might celebrate a win on newsletter sign-ups, unaware it's likely a statistical ghost. To fix this, they could use a Bonferroni correction, adjusting the required p-value for each test to 0.05 / 20 = 0.0025.