Twyman's Law: Interesting Data is Usually Wrong

THE MENTAL MODEL: Twyman's Law is a principle of data analysis that urges skepticism towards any figure that looks particularly interesting or unusual. The core idea is that the more surprising the data, the more likely it is the result of an error in measurement, collection, or processing, rather than a reflection of a real-world phenomenon. It's a professional's version of the saying, "If it looks too good to be true, it probably is."

HOW IT WORKS: This law is a heuristic, not a formula. When an analyst encounters an outlier—a data point that deviates significantly from the norm—their first step should be to investigate the data's lineage, not to explain the phenomenon. This means checking the tracking code, database queries, and any data transformation jobs. The working assumption is that an error is the most probable cause for any dramatic deviation. Only after all potential sources of error have been ruled out should one begin to explore a real-world explanation for the interesting figure.

WHEN TO USE IT: This law applies universally in any field involving data analysis, especially business intelligence, A/B testing, and system monitoring. If a dashboard shows a 50% drop in daily active users overnight, the first hypothesis should not be "half our users quit," but rather "the analytics job probably failed." If an A/B test shows a new button color increases conversions by 200%, suspect a bug in how conversion events are fired before rolling it out.

WHEN NOT TO USE IT: The law is a guide for investigation, not a rule for discarding data. You should not automatically delete outliers. After a thorough investigation proves the data is correct, the outlier might represent a genuine, important event like a viral social media mention or a critical system failure. The law guides your initial reaction and investigative priority; it does not give you permission to ignore data you have validated as accurate just because it is surprising.

ONE CANONICAL EXAMPLE: A marketing team sees a 1000% increase in website traffic from a small country. A junior analyst might get excited, thinking a campaign went viral. Applying Twyman's Law, a senior analyst would immediately suspect a botnet. They would investigate the IP addresses and on-site behavior of the new traffic. In most cases, they'll find it's junk traffic, and the "interesting" figure was just an error in what should be measured.