What does a p-value of 0.03 mean in an A/B test?

WHAT THIS TESTS: This question tests your practical understanding of statistical significance and your ability to apply a decision-making framework (p-value vs. alpha) without misinterpreting the data. It's a check for the basic data literacy required to make sound product decisions, distinguishing rote memorization from applied knowledge.

A GOOD ANSWER COVERS: A complete answer has four parts in order. First, define the p-value correctly: A p-value of 0.03 means there is a 3% probability of observing a result at least as extreme as the one you got, assuming the null hypothesis (that there's no real difference between variants) is true. Second, state the decision rule: You compare the p-value to your pre-determined significance level, alpha. In this case, 0.03 is less than the specified alpha of 0.05. Third, state the statistical conclusion: Because p < alpha, you reject the null hypothesis and conclude the result is statistically significant. Fourth, state the business decision: Based on this, you would typically decide to roll out the new feature, assuming the metric moved in the desired direction.

COMMON WRONG ANSWERS: The most common red flag is misstating the definition. Saying "there's a 3% chance the result is due to random chance" or "there's a 97% chance our feature works" is incorrect. The p-value is calculated assuming the null hypothesis is true; it cannot tell you the probability of the null hypothesis being true or false. Another mistake is focusing only on "significant" without mentioning practical significance—a 0.1% lift in revenue might be statistically significant with enough traffic but not practically meaningful enough to justify the maintenance cost of the new code.

LIKELY FOLLOW-UPS: Expect questions about error types. For example, "What is a Type I error in this context?" (A false positive: rejecting the null hypothesis when it's actually true, meaning you ship a feature that has no real effect). Or, "What if the p-value was 0.06?" (You would fail to reject the null hypothesis. The result is not statistically significant, and you would typically not ship the feature. You might, however, consider running the test longer if you are close to significance and have a strong prior belief in the feature's value).

ONE CONCRETE EXAMPLE: An A/B test for a new checkout button color aims to increase the conversion rate. The null hypothesis is that the new color has no effect. The test runs and yields a p-value of 0.03. With our pre-defined alpha of 0.05, we reject the null hypothesis. This means we have sufficient evidence to conclude the color change has an effect. If the conversion rate for the new color was higher, we would decide to roll it out. The 3% p-value quantifies the risk of being wrong: if the button color truly made no difference at all, we'd still see a result this strong in 3 out of 100 identical tests just by random luck.