How do you determine required sample size for an A/B test?

WHAT THIS TESTS: Whether you understand that sample size is not a guess but a function of four parameters. Interviewers want to see you know the difference between statistical significance and practical significance, and that you can defend an experiment's runtime before spending traffic.

A GOOD ANSWER COVERS: First, baseline conversion rate, which is your current click-through rate on the blue button, often around two to five percent for e-commerce CTAs. Second, minimum detectable effect, the smallest lift that would justify shipping the green button, such as a relative ten percent increase. Third, alpha, the false-positive rate you are willing to accept, conventionally set at five percent. Fourth, power, the probability of detecting a true effect of that size, conventionally eighty percent. With these four numbers, you use a two-proportion z-test power formula or an online calculator to solve for N per variant. A senior candidate also mentions that you should check for sufficient traffic to reach that sample in a reasonable business window and that you may need to adjust for multiple comparisons if you are running several variants.

COMMON WRONG ANSWERS: Guessing a round number like one thousand users per cell without any statistical basis. Running the test until you achieve significance, which inflates the false-positive rate through peeking. Using a one-week window just because sprint cycles are one week, regardless of whether the sample reaches the required power. Forgetting that the lower the baseline rate, the larger the sample needed to detect the same relative lift. Ignoring the minimum detectable effect and instead asking for the sample to detect any difference, which drives N toward infinity.

LIKELY FOLLOW-UPS: How would your sample size change if the baseline CTR drops from five percent to one percent? What if the business can only tolerate a one percent false-positive rate instead of five percent? How do you handle early stopping or interim peeks without breaking the error rate? What would you do if you do not have enough traffic to reach the required sample in one quarter?

ONE CONCRETE EXAMPLE: Suppose the blue button has a four percent baseline CTR and you want to detect a relative fifteen percent lift to 4.6 percent with ninety percent power and a five percent alpha. A power calculation returns roughly six thousand two hundred users per variant. If your page gets ten thousand unique visitors per week, you need about one and a quarter weeks, so you would likely round up to two full weeks to cover day-of-week effects. If you had instead aimed for a fifty percent lift, the required sample would drop to roughly nine hundred per variant, but that effect size may be unrealistic for a button color change.