P-Value: Gauging Surprise, Not Certainty

WHY IT EXISTS To provide a standardized way to assess whether an observed effect in an experiment is likely due to random chance or a genuine underlying phenomenon. Without a formal method, we're just guessing if a change we made actually had an impact or if the results are a fluke.

THE MENTAL MODEL A p-value is a 'surprise meter.' Imagine you're testing a new button color. The 'null hypothesis' is that the color makes no difference to clicks. The p-value tells you: if that null hypothesis is true, what are the odds you'd see a click difference as big as the one you just measured, just from random luck? A low p-value means 'very surprising,' suggesting the null hypothesis might be wrong.

HOW IT WORKS In null-hypothesis significance testing, you start by assuming there is no effect. After collecting data, you calculate the p-value. This value is the probability of obtaining test results at least as extreme as the ones you actually observed, under the assumption that the null hypothesis is correct. A very small p-value means that such an extreme outcome would be very unlikely if there were truly no effect.

WHEN TO USE IT Use p-values in quantitative fields when you need to perform null-hypothesis significance testing. This is common practice in academic publications and A/B testing to determine if an observed result is statistically significant or could be explained by random noise.

WHEN NOT TO USE IT Do not use a p-value as the sole indicator of a result's importance. A statistically significant result (low p-value) is not necessarily a practically significant one. With a large enough sample size, you can find a tiny p-value for a trivial, unimportant effect. It also does not tell you the probability of your hypothesis being true.

ONE CANONICAL EXAMPLE You run an A/B test on a checkout button, with version A (current) and version B (new design). The null hypothesis is that the new design has no effect on clicks. After the test, version B shows a 2% higher click rate. You calculate a p-value of 0.03. This means that if the button design truly had no effect, there would only be a 3% chance of seeing a click-rate difference of 2% or more, just due to random variation. Because this is unlikely (typically below a 0.05 threshold), you reject the null hypothesis.