What does a p-value of 0.03 mean in an A/B test?

WHAT THIS TESTS: This is a filter question to verify your foundational knowledge for data-driven decisions. Interviewers are testing your ability to correctly define a core concept (p-value), apply a simple decision rule (compare to alpha), and communicate the business implication (the feature likely had an effect). It's a test of precision in your language.

A GOOD ANSWER COVERS: A senior-level answer is concise and precise, hitting four points. First, define the null hypothesis: "The new feature has no effect on the target metric." Second, define the p-value in that context: "A p-value of 0.03 means there is a 3% chance of observing a result at least this extreme, assuming the null hypothesis is true (i.e., assuming the feature had no real effect)." Third, state the decision rule: "We compare the p-value to our pre-defined significance level, alpha, which is 0.05." Fourth, state the conclusion: "Since 0.03 is less than 0.05, we reject the null hypothesis and conclude the result is statistically significant."

COMMON WRONG ANSWERS: The most common red flag is an incorrect definition of the p-value. Candidates often say, "It's the probability that the null hypothesis is true" or "There's a 3% chance the result was random." Both are wrong. The p-value assumes the null is true and tells you how surprising your result is. Another mistake is conflating statistical significance with practical significance. A senior candidate should hint at this nuance, noting that a significant result might not be large enough to be meaningful for business goals.

LIKELY FOLLOW-UPS: Expect questions that probe the nuances. "What if the p-value was 0.06?" (You fail to reject the null; the result is not statistically significant at an alpha of 0.05). "What are the risks of setting alpha too high or too low?" (Too high, e.g., 0.10, increases Type I errors/false positives. Too low, e.g., 0.01, increases Type II errors/false negatives and requires more statistical power). "What other metrics would you look at besides the p-value?" (Effect size, confidence intervals, practical significance).

ONE CONCRETE EXAMPLE: Imagine testing a new checkout button. The null hypothesis is that the new button has no effect on conversion rate. We run the test and get a p-value of 0.03. This means that if the button truly had no effect, there would only be a 3% probability of seeing the conversion rate difference we observed (or an even larger one) just by random chance. Since our alpha was 0.05, our threshold for being "surprised" by a result, and 3% is less than 5%, we reject the idea that the button had no effect. We conclude the new button likely caused the change in conversion.