Hypothesis Testing: A Courtroom for Your Data

WHY IT EXISTS: To move beyond anecdotal evidence or gut feelings when making decisions. Statistical hypothesis testing provides a formal, repeatable framework for evaluating whether an observed effect in data is likely a real pattern or just due to random chance.

THE MENTAL MODEL: Think of it as a courtroom trial for a claim. The "null hypothesis" (H0) is the defendant, presumed innocent until proven guilty. This is the default state, like "this new feature has no effect." The "alternative hypothesis" (H1) is the prosecutor's claim: "this feature increases engagement." Your data is the evidence presented to the court.

HOW IT WORKS: The process has five main steps. First, you state your null (H0) and alternative (H1) hypotheses. Second, you choose a significance level (alpha), typically 0.05, which is your threshold for "reasonable doubt." Third, you collect data and calculate a test statistic, which summarizes how far your sample data deviates from the null hypothesis. Fourth, you compute the p-value from this statistic. The p-value is the probability of seeing your results, or even more extreme ones, if the null hypothesis were actually true. Fifth, you compare your p-value to alpha. If p < alpha, you "reject the null hypothesis." If p >= alpha, you "fail to reject" it.

WHEN TO USE IT: Use hypothesis testing when you need to make a decision between two competing possibilities based on sample data. It's the engine behind A/B testing a website change, determining if a new drug is more effective than a placebo, or checking if a manufacturing change has altered product quality.

WHEN NOT TO USE IT: It's not for exploratory analysis where you're just looking for patterns without a pre-defined question. Crucially, you never "accept" the null hypothesis; you only fail to find enough evidence to reject it—an absence of evidence is not evidence of absence. Also, avoid it if the assumptions of your chosen statistical test (like data normality or sample size) are not met.

ONE CANONICAL EXAMPLE: Imagine an A/B test for a new checkout button. The null hypothesis (H0) is that the new button has the same conversion rate as the old one. The alternative (H1) is that the conversion rate is different. You set your significance level alpha to 0.05. After running the test, you find the new button has a higher conversion rate and your statistical test yields a p-value of 0.03. Since 0.03 is less than 0.05, you reject the null hypothesis. You have statistically significant evidence to conclude the new button's performance is different from the old one.