Type I vs. Type II Errors: False Alarms vs. Missed Detections

The mental model

Think of a smoke alarm. A Type I error is a false positive: the alarm blares because of burnt toast, but there's no fire. You acted on a false signal. A Type II error is a false negative: the house is filling with smoke, but the alarm stays silent. You missed a critical signal.

How it works

In statistics, we start with a "null hypothesis"—the default assumption that nothing interesting is happening (e.g., "this new drug has no effect").

A Type I error (false positive) is incorrectly rejecting a true null hypothesis. Your data shows a significant effect just by random chance, so you conclude the drug works when it doesn't. You've found a ghost in the data.

A Type II error (false negative) is incorrectly failing to reject a false null hypothesis. The drug really does work, but your test wasn't powerful enough to detect it. You miss a real discovery.

These errors are a trade-off. If you make your test extremely sensitive to avoid missing any effect (avoiding Type II errors), you increase the risk of reacting to random noise (more Type I errors). If you make your test very strict to avoid false alarms (avoiding Type I errors), you increase the risk of missing a real, smaller effect (more Type II errors).

When to use it

This concept is a lens for evaluating any system that makes a decision under uncertainty. Three places this shows up: first, in A/B testing, where a Type I error means shipping a useless feature, and a Type II error means missing a real improvement; second, in medical screening, where a false positive causes anxiety and a false negative means a disease goes untreated; third, in manufacturing quality control, where a false positive means scrapping a good batch, and a false negative means shipping defective products.

When not to use it

The framework always applies, but the priority changes based on consequences. You prioritize avoiding Type I errors (false positives) when the cost of a false alarm is high, such as convicting an innocent person or recalling a safe product. You prioritize avoiding Type II errors (false negatives) when the cost of missing something is catastrophic, such as in airport security screening or testing for a fatal but treatable disease. It's better to have many inconvenient false alarms than to miss one real threat.

One canonical example

The fable of "The Boy Who Cried Wolf" illustrates the trade-off. The villagers' null hypothesis is "there is no wolf." The boy's first cries are Type I errors (false positives), causing the villagers to waste time and lose trust. After several false alarms, they adjust their standards and ignore his cries. When a real wolf appears, their failure to respond is a catastrophic Type II error (a false negative).