Explain the difference between statistical and practical significance

WHAT THIS TESTS: This question checks whether you treat experimentation as a business decision tool rather than a purely mathematical exercise. Interviewers want to see that you know a low p-value only tells you an effect is unlikely to be noise; it does not tell you if the effect is large enough to justify engineering time, maintenance, or user friction. Senior candidates are expected to connect sample size, effect size, and cost-benefit analysis in their reasoning.

A GOOD ANSWER COVERS: First, define statistical significance as the probability that the observed effect is real and not due to chance, typically assessed with a p-value threshold like 0.05. Second, define practical significance as whether the magnitude of that effect, the effect size, matters in the real world given the context. Third, explain the sample size trap: with millions of users, even a 0.1 percent lift can reach statistical significance because the standard error shrinks, but the absolute business impact may be tiny. Fourth, describe a clear decision framework that weighs the lift against implementation cost, opportunity cost, and technical debt.

COMMON WRONG ANSWERS: A major red flag is saying that a p-value below 0.05 automatically means you should ship the feature. Another weak response is ignoring effect size entirely and only talking about confidence intervals or p-values. Some candidates also conflate practical significance with sample size, arguing that small samples are the only reason to care about practical significance, which misses the point. Finally, giving a vague example without numbers, such as saying a change improved metrics but was not important, sounds hand-wavy; senior answers use concrete percentages and dollar or hour estimates.

LIKELY FOLLOW-UPS: The interviewer may ask how you would set a minimum detectable effect before launching an experiment, or how you balance statistical power with runtime costs. They might also probe whether you would ship a feature with a large effect size that is not statistically significant, which tests your understanding of Type I and Type II errors in business context. Another common thread is how you communicate a statistically significant but practically insignificant result to a product manager who is emotionally invested in shipping.

ONE CONCRETE EXAMPLE: Imagine you run an A/B test on a checkout flow with ten million users. The variant shows a 0.05 percent increase in conversion with a p-value of 0.01, so the result is statistically significant. However, the variant requires adding a new microservice that costs fifty thousand dollars a year to maintain and adds two hundred milliseconds of latency. The absolute revenue gain from the lift is projected at twenty thousand dollars a year. Because the engineering cost and latency penalty outweigh the tiny gain, the result is not practically significant and you should not ship it.

Source: statsig.com