Interpret a p-value in an A/B test

WHAT THIS TESTS The question separates people who can recite a definition from those who genuinely understand it. P-values are widely misinterpreted, so the interviewer listens for what you correctly include and, just as important, what you correctly exclude.

A GOOD ANSWER COVERS Define the p-value precisely: assuming the null hypothesis that the two variants have the same true conversion rate, it is the probability of observing a difference at least as extreme as the one you measured. For a p-value of 0.03, that means if there were genuinely no difference, random sampling would produce a gap this large or larger only about 3 percent of the time. Since 0.03 is below a conventional 0.05 significance level, you reject the null and call the result statistically significant. Stress that significance is not the same as importance: a tiny lift can be significant with enough traffic, so always report the effect size and confidence interval alongside the p-value, and consider practical or business significance.

COMMON WRONG ANSWERS Saying there is a 3 percent probability the result happened by chance or a 97 percent probability the variant is better, both of which invert the conditional. Treating a low p-value as proof of a large or valuable effect. Forgetting that peeking and multiple comparisons inflate false positives.

LIKELY FOLLOW-UPS What is the difference between a p-value and a confidence interval? How does peeking at results early inflate the false-positive rate? What is statistical power and how does it relate to the threshold?

ONE CONCRETE EXAMPLE Variant B shows a conversion lift over A with a p-value of 0.03. Correct reading: if A and B were truly identical, you would see a lift this big only 3 times in 100 by chance, so you act on it. Incorrect reading you must avoid: there is a 97 percent chance B is the better variant. The data do not support that statement.