What is a p-value in A/B tests and what does threshold mean?

WHAT THIS TESTS: This question checks whether you understand the mechanics of frequentist hypothesis testing or if you are just clicking buttons in an experimentation platform. The interviewer wants to see that you can define a p-value precisely, distinguish it from the probability of a hypothesis being true, and explain the alpha threshold as a decision rule rather than a property of the data.

A GOOD ANSWER COVERS: A strong answer hits four things in order. First, define the p-value as the probability of obtaining a result at least as extreme as the one observed, given that the null hypothesis is true. Second, clarify that in an A/B test the null hypothesis usually states there is no difference between the control and variant. Third, state that a common threshold is 0.05 and that this is called the significance level or alpha. Fourth, explain that the threshold means you are willing to reject the null hypothesis five percent of the time when it is actually true, which is the false-positive rate.

COMMON WRONG ANSWERS: The biggest red flag is saying the p-value is the probability that the variant is better than control, or that there is a ninety-five percent chance the result is real. Another red flag is describing the threshold as the probability that the test is wrong. Some candidates also confuse the p-value with the false discovery rate or say it measures effect size.

LIKELY FOLLOW-UPS: An interviewer might ask how the p-value relates to statistical power and sample size. They might also ask what happens if you run many tests or peek at results repeatedly. A sharp follow-up is asking why a five percent threshold is standard and when you would choose a stricter one.

ONE CONCRETE EXAMPLE: Imagine you run an A/B test on a checkout button and observe a two percent lift in conversion. The p-value is 0.03. This means that if the new button actually has no effect, there is a three percent chance you would see a two percent lift or larger purely due to random variation. Setting alpha at 0.05 means you decided beforehand to call such a result statistically significant and risk a false positive in five percent of tests where the null is true.