Hypothesis Testing: Is Your Data Signal or Noise?

The mental model

Statistical hypothesis testing is like a courtroom trial for your data. You begin by assuming a default state of 'innocence,' called the null hypothesis (H0), which typically represents 'no effect' or 'no difference.' Your data acts as the evidence. The goal is to see if this evidence is strong enough to 'convict' and reject the null hypothesis in favor of an alternative hypothesis (H1).

How it works

The process involves a few key steps. First, you state your null (e.g., 'the new button does not increase clicks') and alternative hypotheses. Second, you collect data from an experiment, like an A/B test. Third, you calculate a 'test statistic' (like a t-score or z-score) that summarizes how much your data deviates from the world described by the null hypothesis. Fourth, you compute a p-value from this statistic. The p-value is the probability of observing data at least as extreme as yours, assuming the null hypothesis is true. Finally, you compare your p-value to a pre-set significance level (alpha, often 0.05). If p < alpha, you reject the null hypothesis and declare the result 'statistically significant.'

When to use it

Use hypothesis testing when you need to make a decision based on sample data and want to quantify your confidence. Three common places this appears: first, in A/B testing to decide if a new feature improves a key metric; second, in manufacturing to verify if a process change reduced defects; third, in clinical trials to determine if a new drug is more effective than a placebo.

When not to use it

Avoid hypothesis testing for open-ended data exploration. Running many tests on the same dataset until you find a low p-value is an invalid practice called 'p-hacking' or 'data dredging.' Also, remember that statistical significance is not the same as practical significance. With a large enough sample, a tiny, meaningless effect (e.g., a 0.01% conversion lift) can become statistically significant, but it may not be worth implementing in the real world.

One canonical example

A website tests a new headline. The null hypothesis is that the new headline has no effect on user engagement time. After running the test, they calculate a p-value of 0.02. Since this is less than their chosen alpha of 0.05, they reject the null hypothesis. They conclude there is a statistically significant difference in engagement time and can confidently roll out the new headline.