Bonferroni Correction: Raising the Bar for Significance

WHY IT EXISTS The multiple comparisons problem is a statistical trap. A 5% chance of a false positive (a p-value < 0.05) might be acceptable for a single experiment. But if you run 20 tests, your chance of getting at least one false positive by pure random luck balloons to over 64%. You start seeing significant results everywhere, even in random noise. The Bonferroni correction was created to prevent this.

THE MENTAL MODEL Think of it as an anti-lottery ticket. The more tests you run, the higher your chance of hitting a "false positive jackpot." The Bonferroni correction makes the price of each ticket (the significance threshold) much higher. If you want to run 10 tests, you have to be 10 times as certain about any single result before you can declare it a winner. This reduces the chance of being fooled by randomness.

HOW IT WORKS The math is simple division. Take your desired significance level, called alpha (usually 0.05), and divide it by the number of tests you are running, 'n'. The result, alpha / n, becomes your new, much stricter p-value threshold. For any single test to be considered significant, its p-value must be below this new corrected value.

WHEN TO USE IT Use it for a small number of pre-planned statistical tests where you need to strictly control the family-wise error rate—the probability of making even one false positive. Its simplicity makes it easy to apply and explain, which is a benefit when communicating results to non-statisticians.

WHEN NOT TO USE IT Avoid it for a large number of comparisons. It becomes so conservative that it dramatically increases your chance of a false negative (missing a real effect). If you test 1,000 variants, the correction makes it nearly impossible to find a significant result. In these cases, methods that control the False Discovery Rate (FDR), like the Benjamini-Hochberg procedure, are more powerful and appropriate.

ONE CANONICAL EXAMPLE You are A/B/C/D testing four new button colors against the original. This means you are running four comparisons (New A vs. Control, New B vs. Control, etc.). Your team's standard for significance is p < 0.05. Applying a Bonferroni correction, your new significance threshold becomes 0.05 / 4 = 0.0125. A variant that shows a p-value of 0.03, which might have seemed promising, is now correctly identified as not statistically significant under this stricter regime.