What is a p-value? Interpret p = 0.03 at alpha = 0.05.

WHAT THIS TESTS: This question checks whether you can define a p-value precisely in the context of null-hypothesis significance testing and interpret it relative to a significance level without falling into popular misinterpretations. Interviewers care that you treat the p-value as a statement about data under a model rather than as a direct probability about hypotheses or real-world truth.

A GOOD ANSWER COVERS: A strong answer has two parts. First, define the p-value as the probability of obtaining test results at least as extreme as the result actually observed, assuming the null hypothesis is correct. Second, interpret the specific numbers by comparing the p-value to alpha. Since 0.03 is less than 0.05, you would reject the null hypothesis at the 5% significance level. A senior candidate might add nuance by noting that the 0.05 threshold is arbitrary, that p-values do not measure effect size or practical importance, and that the result is one observation in a broader analytical process.

COMMON WRONG ANSWERS: The most dangerous error is saying the p-value is the probability that the null hypothesis is false or the probability that the result occurred by chance. Another red flag is claiming there is a 97% probability that the alternative hypothesis is true. Some candidates also confuse the significance level alpha with the probability of making a Type I error after the fact, or they imply that 0.03 means the finding is highly important or replicated. These mistakes signal a rote rather than conceptual understanding.

LIKELY FOLLOW-UPS: An interviewer might ask how you would handle a p-value of 0.06, what a confidence interval adds that a p-value does not, or how you would correct for multiple comparisons. They may also probe the difference between statistical significance and practical significance, or ask about the assumptions required for the p-value to be valid.

ONE CONCRETE EXAMPLE: Suppose you are testing whether a new checkout flow increases conversion. The null hypothesis is that the conversion rate is unchanged. You run an experiment and obtain a p-value of 0.03. This means that if the new flow actually had no effect, there is a 3% probability of observing a conversion difference as large as or larger than the one you measured purely from sampling variation. Because 3% is below your 5% alpha threshold, you reject the null hypothesis and conclude there is statistically significant evidence of a change, though you would still want to check the effect size and business impact before launching.