Determine A/B test sample size

WHAT THIS TESTS The question checks whether you know that sample size is determined by four interacting statistical parameters, and whether you can reason about the business cost of tightening each one.

A GOOD ANSWER COVERS Name the four inputs. The baseline conversion rate is the current rate of the control. The minimum detectable effect is the smallest lift worth catching, here 2 percent, expressed as absolute or relative and stated clearly. The significance level alpha is the tolerated false-positive rate, conventionally 0.05, controlling how often you wrongly declare a winner. The statistical power, one minus beta, conventionally 0.8, is the probability of detecting a real effect of the chosen size, controlling false negatives. The relationships: required sample size grows as the MDE shrinks (small effects are hard to see), as alpha is lowered (more certainty demanded), and as power is raised (fewer missed wins), and it also depends on the baseline rate's variance.

COMMON WRONG ANSWERS Forgetting power entirely and only mentioning significance, treating the MDE as a fixed property rather than a business choice, ignoring that the baseline rate affects variance, or assuming you can simply run until results look good, which inflates false positives.

LIKELY FOLLOW-UPS What happens if you stop the test early when it looks significant? How do one-sided versus two-sided tests change the number? How do you handle multiple metrics or variants? What if traffic is too low to ever reach the needed sample?

ONE CONCRETE EXAMPLE Detecting a 2 percent lift on a 10 percent baseline at alpha 0.05 and 80 percent power requires far more users than detecting a 10 percent lift, because smaller effects are noisier to confirm. The business trade-off is stark: insisting on catching a 2 percent gain might mean running the test for many weeks, delaying the launch decision, whereas accepting a larger MDE shortens the test but risks shipping nothing when the true gain is small but real.