Confidence intervals versus p-values, explained simply
statistical literacy for A/B decisions.
define a CI as a plausible range for the true effect with its precision, contrast it with a binary p-value, and read [-1%, 7%] as inconclusive because it spans zero.
What's really being asked
The interviewer checks both correct statistical understanding and the ability to translate it into a clear product decision, since misreading intervals leads to shipping non-effects.
The full answer
Define a confidence interval as a range of plausible values for the true effect, computed so that the procedure captures the true value in ninety-five percent of repeated experiments. Crucially it conveys both the direction and magnitude of an effect and how precise the estimate is: a narrow interval means a well-pinned estimate, a wide one means high uncertainty. Contrast this with a p-value, which reduces everything to a single threshold answer about whether the effect differs from zero, telling you nothing about how big or how precise. Then read the given interval: [-1%, 7%] includes zero, so the result is not statistically conclusive; the true effect could be a small negative, nothing, or up to seven percent positive. Its width signals the sample is too small to decide, so the honest recommendation is to gather more data rather than ship on a hunch.
The mistakes people make
Stating there is a ninety-five percent probability the true value lies in this specific interval, which is a frequentist misinterpretation. Treating any positive point estimate as a win despite the interval crossing zero. Saying a non-significant result proves no effect, when it may just be underpowered.
What usually comes next
How does sample size change interval width. What is the difference between statistical and practical significance. How would you decide a minimum detectable effect up front. Why can a p-value below the threshold still represent a trivial effect.
A concrete example
To the product manager: our best guess is a three percent lift, but the data is consistent with anything from a one percent drop to a seven percent gain, so we cannot confidently say the change helps. Because the range crosses zero, we should not ship yet; running the test longer to collect more samples will narrow the range and let us decide.
Interview question
An A/B test on a new checkout flow reports a 95% confidence interval of -1% to 7% lift. What is the correct interpretation?
- a.There is a 95% probability the true lift falls between -1% and 7%, so the team should ship the change with high confidence.
- b.Since the point estimate is positive, roughly 3%, the change is working and can ship immediately.
- c.The interval spans zero, so the result does not rule out no effect or a small harm, and the wide range signals more data is needed before deciding.Correct
- d.The result is not statistically significant, which proves the new checkout flow has no real effect on conversion.
Why? this is the answer
The interval crosses zero, so the true effect could be a small loss or as much as a 7% gain, meaning the data does not support a confident decision either way. Option A is the classic frequentist misreading the card warns against, treating the interval as a probability statement about this one range.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #statistics
- #confidence-intervals
- #ab-testing
- #data-literacy
- #experimentation
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on statistics — each one lists the topics its interview covers.
See open roles