A/B Testing: Calculating Your Sample Size

Why it exists

Running an A/B test on too few users is like flipping a coin ten times and declaring it biased if you get seven heads. Random chance can easily create misleading results. Sample size determination was created to protect against making important decisions based on this statistical noise, ensuring that an observed effect is likely real.

The mental model

Think of sample size calculation as setting the rules for a fair race before it starts. You decide beforehand how long the race needs to be (the sample size) for a win to be meaningful. You don't stop the race early just because one runner takes a brief lead. This prevents you from declaring a winner based on a lucky start rather than true, sustained performance.

How it works

You don't typically do the math by hand; you use a calculator. But you need to provide the key ingredients. First, the baseline conversion rate: the performance of your current version. Second, the minimum detectable effect (MDE): the smallest improvement you'd consider meaningful. Third, statistical significance (alpha, usually 95%): your risk of a false positive. Fourth, statistical power (1-beta, usually 80%): your chance of detecting a real effect if it exists. The calculator uses these to tell you how many users you need in each group.

When to use it

Always perform this calculation before launching a controlled experiment comparing two or more variants. This applies to testing changes in user interfaces, marketing copy, algorithms, or any other user-facing experience. It is a foundational step in A/B testing, A/A testing (to validate your tools), and more complex multivariate tests.

When not to use it

Sample size calculation is for hypothesis testing, not for general exploration. You don't need it for qualitative feedback like user interviews, usability testing, or when you are simply monitoring metrics after a full launch. If a change is so subtle that it requires an enormous sample size, it may not be worth testing at all.

One canonical example

A SaaS company has a signup page with a 5% conversion rate. They want to test a new headline and decide a 0.5% absolute improvement (to 5.5%) is the minimum lift they care about. With 95% significance and 80% power, an online calculator would tell them they need about 25,000 users for the control group and 25,000 for the variant group. Running the test on only 5,000 users would risk missing a real win or declaring a random fluctuation as a success.