Null Hypothesis: Assume No Effect Until Proven Otherwise
The null hypothesis is your default assumption: nothing changed. You run experiments to gather enough evidence to reject this default. In A/B testing, the null is that your new feature has no effect, while the alternative is that it does.
WHY IT EXISTS The null hypothesis framework was created to bring scientific rigor to testing claims. It protects us from confirmation bias—the tendency to see patterns in random noise. By forcing us to assume there is no effect by default, it sets a high bar for proving that a change actually made a difference.
THE MENTAL MODEL Think of a court trial. The null hypothesis is the defendant, presumed innocent (meaning, no effect occurred). The alternative hypothesis is the prosecution's claim that the defendant is guilty (an effect did occur). Your experimental data is the evidence. You need to present enough evidence to convince the jury 'beyond a reasonable doubt' (i.e., reach statistical significance) to reject the presumption of innocence.
HOW IT WORKS You start by defining two opposing statements. The null hypothesis (H₀) states there is no difference between groups (e.g., 'the new button has the same click rate as the old one'). The alternative hypothesis (H₁) states there is a difference. After collecting data, you perform a statistical test to determine the probability (the p-value) that you'd see your results if the null hypothesis were true. If this probability is very low (typically under 5%), you reject the null hypothesis.
WHEN TO USE IT This is the foundation of all statistical hypothesis testing, especially A/B testing. Use it whenever you need to determine if an observed effect is real or just due to random chance. It's for answering questions like, 'Did this new feature actually increase engagement?' or 'Is this new algorithm truly faster?'.
WHEN NOT TO USE IT This formal framework isn't necessary for purely exploratory analysis where you're just looking for interesting patterns without a specific question. It's also overkill if an effect is so large and obvious that the outcome is certain without statistical validation.
ONE CANONICAL EXAMPLE An engineering team wants to know if a new caching strategy reduces API response time. Null Hypothesis (H₀): The average response time with the new cache is the same as the old cache. Alternative Hypothesis (H₁): The average response time with the new cache is different from the old cache. After a test, they find the new cache is faster and the result has a p-value of 0.01. Since 0.01 is less than their significance threshold of 0.05, they reject the null hypothesis and conclude the new caching strategy has a statistically significant effect on performance.
Read the original → en.wikipedia.org
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.