Statistical Significance: Is Your Result Real or Just Random?

THE MENTAL MODEL: Statistical significance is a formal procedure to rule out random chance as the explanation for your data. You start by assuming your change has no effect at all (the "null hypothesis"). Then, you calculate how likely your observed data is under that assumption. If the data is very unlikely to occur by chance, you reject the "no effect" assumption and conclude your change had a real effect.

HOW IT WORKS: The process compares two numbers: the p-value and the significance level (alpha). First, you state a null hypothesis, like "this new button color has no effect on clicks." Second, you choose a significance level (alpha, α), which is your tolerance for being wrong. A typical alpha is 0.05, meaning you accept a 5% risk of a false positive. Third, after running your experiment, you calculate a p-value. The p-value is the probability of seeing your result (or one even more extreme) if the null hypothesis were true. Finally, if your p-value is less than alpha (p < α), you declare the result statistically significant and reject the null hypothesis.

WHEN TO USE IT: Use this framework whenever you make a decision based on a sample of data, not the whole population. It is the standard for interpreting A/B tests for product changes, analyzing marketing campaign effectiveness, or validating scientific experiments. It provides a disciplined way to avoid acting on random noise.

WHEN NOT TO USE IT: Statistical significance is not a measure of importance or effect size. A tiny, practically meaningless improvement (like a 0.01% conversion lift) can be statistically significant if you have enough data. Do not use it as your only decision-making tool; always consider the magnitude of the effect and its business impact. Also, it's for samples; if you have data for your entire population, you don't need to infer an effect—you can measure it directly.

ONE CANONICAL EXAMPLE: You A/B test a new signup button. The null hypothesis is that the new button has no effect on signups. You set your significance level α to 0.05. After the test, you calculate a p-value of 0.03. Since 0.03 is less than 0.05, you reject the null hypothesis and conclude the new button has a statistically significant effect. If the p-value had been 0.10, you would fail to reject the null hypothesis, meaning you don't have enough evidence to claim the new button is better.