Type I vs. Type II Errors: False Alarms vs. Missed Detections

THE MENTAL MODEL: Think of a smoke alarm. A Type I error is a false positive: the alarm blares because of burnt toast, but there's no fire. You acted on a false signal. A Type II error is a false negative: the house is filling with smoke, but the alarm stays silent. You missed a critical signal.

HOW IT WORKS: In statistics, we start with a "null hypothesis"—the default assumption that nothing interesting is happening (e.g., "this new drug has no effect").

A Type I error (false positive) is incorrectly rejecting a true null hypothesis. Your data shows a significant effect just by random chance, so you conclude the drug works when it doesn't. You've found a ghost in the data.

A Type II error (false negative) is incorrectly failing to reject a false null hypothesis. The drug really does work, but your test wasn't powerful enough to detect it. You miss a real discovery.

These errors are a trade-off. If you make your test extremely sensitive to avoid missing any effect (avoiding Type II errors), you increase the risk of reacting to random noise (more Type I errors). If you make your test very strict to avoid false alarms (avoiding Type I errors), you increase the risk of missing a real, smaller effect (more Type II errors).

WHEN TO USE IT: This concept is a lens for evaluating any system that makes a decision under uncertainty. Three places this shows up: first, in A/B testing, where a Type I error means shipping a useless feature, and a Type II error means missing a real improvement; second, in medical screening, where a false positive causes anxiety and a false negative means a disease goes untreated; third, in manufacturing quality control, where a false positive means scrapping a good batch, and a false negative means shipping defective products.

WHEN NOT TO USE IT: The framework always applies, but the priority changes based on consequences. You prioritize avoiding Type I errors (false positives) when the cost of a false alarm is high, such as convicting an innocent person or recalling a safe product. You prioritize avoiding Type II errors (false negatives) when the cost of missing something is catastrophic, such as in airport security screening or testing for a fatal but treatable disease. It's better to have many inconvenient false alarms than to miss one real threat.

ONE CANONICAL EXAMPLE: The fable of "The Boy Who Cried Wolf" illustrates the trade-off. The villagers' null hypothesis is "there is no wolf." The boy's first cries are Type I errors (false positives), causing the villagers to waste time and lose trust. After several false alarms, they adjust their standards and ignore his cries. When a real wolf appears, their failure to respond is a catastrophic Type II error (a false negative).