Determine A/B test sample size
WHAT IT TESTS: the inputs to a power calculation. OUTLINE: define baseline rate, minimum detectable effect, significance (alpha), and power (1-beta); smaller effects and stricter thresholds need more users. RED FLAG: ignoring power or treating MDE as fixed.
WHAT IT TESTS: whether you understand the levers behind sample size and their business cost. ANSWER OUTLINE: you must define the baseline conversion rate, the minimum detectable effect (the 2% lift), the significance level alpha (false-positive rate), and statistical power 1-beta (chance of catching a real effect). Smaller MDE, lower alpha, and higher power all increase required sample size. The business trade-off: detecting tiny lifts or demanding high confidence means longer tests and delayed decisions.
Read the original → interview
- #sample-size
- #statistical-power
- #ab-testing
- #experimentation
- #significance
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.