A/B test: 0.1% lift. Statistical vs practical significance?

WHAT THIS TESTS: This question tests whether you can separate statistical inference from product judgment. A senior engineer must know that a low p-value only tells you the observed effect is unlikely under the null hypothesis; it says nothing about whether the effect is large enough to justify engineering hours, maintenance, or opportunity cost. The interviewer wants to see stakeholder communication skills and quantitative framing.

A GOOD ANSWER COVERS: A strong answer hits four things in order. First, define statistical significance precisely: it is the probability of seeing a result at least as extreme as the 0.1% lift if the null hypothesis were true, typically evaluated against a pre-set alpha like 0.05. Second, define practical significance as an effect size that generates more value than it costs to build and maintain. Third, frame the 0.1% by putting a confidence interval around it and comparing that interval to the minimum detectable effect the experiment was powered for; if the upper bound is still tiny, the result is statistically real but economically irrelevant. Fourth, translate the lift into dollars by multiplying the 0.1% by annual transactions and average order value, then subtract engineering and maintenance cost to show net impact.

COMMON WRONG ANSWERS: Red flags include conflating the two concepts by saying statistically significant means important, insisting the feature must ship because p is less than 0.05, or ignoring the confidence interval entirely. Another weak pattern is answering with pure philosophy and no numbers; senior candidates should show a back-of-the-envelope revenue estimate. Finally, failing to mention opportunity cost is a signal of immature product thinking.

LIKELY FOLLOW-UPS: The interviewer may ask how you would design the next experiment to detect a smaller effect, what you would do if the confidence interval included zero, or how you would communicate a no-ship decision to the product manager. They may also ask about Bayesian alternatives or how to set a minimum detectable effect before launch.

ONE CONCRETE EXAMPLE: Imagine an e-commerce checkout flow with one million transactions per month and a fifty dollar average order value. A 0.1% lift equals five thousand dollars in monthly revenue or sixty thousand dollars annually. If the feature requires two engineers for one month at a loaded cost of twenty thousand dollars each and adds ongoing maintenance complexity, the net first-year value could be negative. In that case you would tell the PM that the result is statistically real but practically negligible, and you recommend shelving the feature or bundling it with a larger initiative.