Explain statistical power, MDE, and sample size impact

WHAT THIS TESTS: This question probes whether you understand the fundamental frequentist relationship between three levers in experiment design: statistical power, minimum detectable effect, and sample size. Interviewers want to see that you treat sample size as a derived quantity rooted in statistical constraints, not an arbitrary business input. They also care if you can explain the practical cost of getting this wrong: underpowered experiments burn user traffic while yielding inconclusive results.

A GOOD ANSWER COVERS: First, define statistical power as the probability that a test correctly rejects the null hypothesis given that a prespecified effect actually exists. Second, define MDE as the smallest effect size the experiment is designed to reliably detect at that power level. Third, explain the directional relationship: for a fixed significance level, increasing power or shrinking the MDE both require a larger sample size because you are asking the test to resolve a smaller signal from the same noise. Fourth, mention that this is why teams fix two of the three variables, typically power at eighty percent and alpha at five percent, then solve for sample size.

COMMON WRONG ANSWERS: A major red flag is treating sample size as a gut feel or budget decision divorced from power and MDE. Another is conflating power with significance level; candidates sometimes say higher power means more false positives, when it actually concerns false negatives. A third error is ignoring the business implication: claiming you can detect a half-percent lift with ninety-five percent power on a thousand users reveals a lack of numerical intuition for variance.

LIKELY FOLLOW-UPS: The interviewer may ask how you would estimate variance for a binomial metric like conversion rate. They might also ask what you would do if the required sample size exceeds your monthly traffic, which tests whether you know to relax power, raise MDE, or extend runtime rather than running an underpowered test. A third follow-up is how sequential testing or peeking affects power calculations.

ONE CONCRETE EXAMPLE: Imagine a checkout flow A/B test with a baseline conversion rate of ten percent. If you want eighty percent power to detect a relative lift of five percent, meaning an absolute move from ten percent to ten point five percent, a standard two-proportion z-test requires roughly sixty thousand users per variant. If the business insists on detecting a two percent relative lift instead, the MDE shrinks and the required sample balloons to roughly three hundred seventy thousand users per variant. If traffic is capped, you must either accept a higher false-negative rate by lowering power to sixty percent or accept a larger MDE.