P-value: Probability of Your Data, Not Your Hypothesis

THE MENTAL MODEL: A p-value is a measure of surprise. It answers the question: assuming my default theory (the "null hypothesis") is true, how likely is it that I would see data this extreme just by random chance? A very small p-value means your observed result would be very surprising if the null hypothesis were true, which in turn suggests the null hypothesis might be wrong.

HOW IT WORKS: First, you state a null hypothesis (H0), which is the default, "no effect" scenario. For example, "This new feature does not change user engagement." Then you collect data from an experiment. The p-value is the calculated probability of obtaining your observed data (or more extreme data) if H0 were actually true. If this probability is smaller than a predetermined threshold, called the significance level (alpha, commonly 0.05), you "reject the null hypothesis." This doesn't prove your alternative hypothesis is true, but it provides evidence against the null.

WHEN TO USE IT: P-values are the foundation of null-hypothesis significance testing, used in controlled experiments to distinguish a real effect from random noise. Three common places you'll see it are: first, in A/B testing to determine if a change to a website or app had a statistically significant impact; second, in scientific research like clinical trials to see if a new drug works better than a placebo; third, in quality assurance to check if a change in a manufacturing process improved the outcome.

WHEN NOT TO USE IT: Do not use a p-value to measure the size or practical importance of an effect. A result can be statistically significant (low p-value) but have a tiny, real-world-irrelevant effect, especially with very large sample sizes. Crucially, a p-value is NOT the probability that the null hypothesis is true or that your alternative hypothesis is false. Misinterpreting it this way is a widespread error.

ONE CANONICAL EXAMPLE: You run an A/B test for a new checkout button. The null hypothesis is that the new button has no effect on conversion rate. After the test, you find the new button increased conversions by 2%, and you calculate a p-value of 0.03. This means that if the button truly had no effect, there would only be a 3% chance of seeing a conversion lift of 2% or more just due to random luck. Since 0.03 is less than the common 0.05 threshold, you reject the null hypothesis and conclude the new button's effect is statistically significant.