What is a p-value, and what does 0.03 practically mean?
This tests your ability to translate stats into business decisions. A great answer defines p-value, compares 0.03 to the standard 0.05 threshold to reject the null hypothesis, and recommends shipping.
What's really being asked
Your understanding of statistical significance and your ability to move from abstract numbers to a concrete business decision. The interviewer wants to see if you can correctly interpret the p-value without falling into common traps, and if you can confidently recommend an action, like shipping a feature, based on the data. It's a test of data literacy, not just memorized statistical definitions.
The full answer
Four key points in order. First, define the p-value correctly: it's the probability of observing your result, or a more extreme one, IF the null hypothesis were true. The null hypothesis is the default assumption that there's no difference between the control and the variant. Second, state the standard practice: we compare the p-value to a pre-determined significance threshold (alpha), which is usually 0.05. Third, apply it to the specific number: since 0.03 is less than 0.05, we declare the result "statistically significant." Fourth, state the practical business outcome: we reject the null hypothesis and conclude the new feature caused the change in conversion. The recommendation is to ship the feature, assuming the conversion lift is meaningful.
The mistakes people make
The biggest red flag is misinterpreting what the p-value represents. Saying "a p-value of 0.03 means there is a 3% chance the result was random" is incorrect. Saying "it means there's a 97% chance the new feature is better" is also incorrect and a very common mistake. This misinterprets the p-value as the probability of the alternative hypothesis being true, when it's actually about the probability of the data given the null hypothesis. Another red flag is not mentioning the null hypothesis or the significance threshold (alpha), which shows a shallow, mechanical understanding.
What usually comes next
"What if the p-value was 0.06?" (Answer: The result is not statistically significant at the 0.05 level, so we fail to reject the null hypothesis; we don't ship). "What is statistical power and why does it matter?" (Answer: Power is the probability of detecting a true effect when one exists; low power increases the chance of a false negative). "What are the risks of shipping with a p-value of 0.03?" (Answer: There's still a small chance it's a false positive, a Type I error).
A concrete example
With a p-value of 0.03, it means that if our new feature actually had NO effect on conversion rate (the null hypothesis), we would only expect to see a result this extreme or more extreme in 3% of identical experiments due to random chance alone. Since 3% is a low probability (less than our 5% threshold), we reject the idea that the feature had no effect. We conclude the observed lift is real and decide to ship the feature to 100% of users.
Interview question
An A/B test on a new feature results in a p-value of 0.03. What is the correct interpretation of this number?
- a.There is a 97% probability that the new feature is better than the original.
- b.If the new feature had no real effect, there is a 3% chance of observing these results or more extreme ones.Correct
- c.The observed result has a 3% probability of being wrong or caused by random chance.
- d.The result is not statistically significant because 3% is less than the standard 5% threshold.
Why? this is the answer
The p-value is the probability of observing the data (or more extreme) assuming the null hypothesis (that there is no effect) is true. Option A is a common misinterpretation; the p-value does not give the probability of the alternative hypothesis being true.
Just read this? Test yourself on what you have been reading.
Read the original → analytics-toolkit.com
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on a/b testing — each one lists the topics its interview covers.
See open roles