Determine Sample Size for a 2% Lift A/B Test
This tests your grasp of statistical power and the business trade-offs in experimentation. A great answer defines baseline conversion rate, minimum detectable effect (MDE), and statistical power. A red flag is ignoring the business context of MDE.
What's really being asked
Your ability to move beyond basic A/B testing definitions and articulate the practical trade-offs between statistical rigor and business velocity. The interviewer wants to see if you can translate a business goal (a 2% lift) into the statistical inputs needed for a valid experiment, and explain the costs and risks associated with those choices.
The full answer
A great answer identifies three, and often four, key parameters. First, the Baseline Conversion Rate (BCR), which is the current performance of the control group, ideally based on at least 30 days of data (e.g., 3.2%). Second, the Minimum Detectable Effect (MDE), which is the smallest lift you care about detecting. For a 2% lift, the MDE is 2%. This is a business decision, not a statistical one. Third, Statistical Significance Level (alpha), the probability of a false positive (Type I error), usually set at 5%. Fourth, Statistical Power (1-beta), the probability of detecting a real effect and avoiding a false negative (Type II error), typically set at 80%. The relationship is that a smaller MDE, lower alpha, or higher power all require a larger sample size.
The mistakes people make
A major red flag is suggesting a fixed time duration, like "run the test for two weeks." This ignores the statistical requirements and leads to underpowered tests. Another common mistake is treating the MDE as a purely statistical value rather than a business decision based on the expected return versus the cost of implementation. For example, is a 2% lift worth the engineering effort? Candidates also often forget to mention power or only mention significance, showing an incomplete understanding of experimental design. Naming only one or two of the parameters is a sign of junior-level knowledge.
What usually comes next
"How would you handle this if you don't have enough traffic to detect a 2% lift in a reasonable timeframe?" (Answer: Increase the MDE, run the test longer, or use alternative methods like sequential testing). "What happens if you 'peek' at the results before the required sample size is reached?" (Answer: It dramatically inflates the false positive rate, leading you to ship changes that have no real effect).
A concrete example
To detect a 2% relative lift on a 3.2% baseline conversion rate (lifting it to 3.264%), with standard 95% significance (alpha=0.05) and 80% power, you would need approximately 393,000 users per variation. If you only get 10,000 users per day to this page, this test would need to run for about 40 days per variation. This illustrates the trade-off: detecting small effects requires a massive sample size and significant time investment.
Interview question
When designing an A/B test, what is the primary consequence of setting a very small Minimum Detectable Effect (MDE), such as a 2% lift?
- a.The risk of a false positive (Type I error) must be increased to make the test feasible.
- b.The test can be concluded more quickly because the success criteria are specific.
- c.A significantly larger sample size is needed to reliably detect such a small change.Correct
- d.The baseline conversion rate is less critical, as the focus is on the relative change.
Why? this is the answer
A smaller MDE requires more statistical evidence to distinguish it from random noise, which necessitates a much larger sample size. Concluding a test early (peeking) is a flawed practice that inflates false positives.
Just read this? Test yourself on what you have been reading.
Read the original → kissmetrics.io
- #a/b testing
- #statistics
- #metrics
- #product analytics
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on a/b testing — each one lists the topics its interview covers.
See open roles