Minimum Detectable Effect: How Small a Change Can You See?

WHY IT EXISTS Experiments cost time and traffic. Running a test that is too small to detect a meaningful change is wasteful, as it will likely conclude 'no effect' even if a real, positive change occurred. MDE was developed to formalize the planning process, ensuring an experiment has a 'good chance of detecting' a worthwhile impact before it even begins.

THE MENTAL MODEL Think of an experiment as a measuring stick and the MDE as the smallest marking on it. If your MDE is 1 inch, you can't use the stick to reliably measure a 1/4-inch object. A smaller MDE means a more sensitive experiment (a ruler with finer markings), but it requires a larger sample size (a longer ruler). You choose your MDE based on the smallest effect that would be practically significant for your business.

HOW IT WORKS MDE is an input for experiment design, not an output of analysis. Before a test, you perform a 'power calculation' that balances four factors: 1. Minimum Detectable Effect (the lift you want to see, like a 1% conversion increase), 2. Statistical Power (the probability of detecting a real effect, usually 80%), 3. Significance Level (the probability of a false positive, usually 5%), and 4. Sample Size. You decide on three of these to calculate the fourth. Typically, you set MDE, power, and significance to find the required sample size.

WHEN TO USE IT Use MDE during the planning phase of any A/B test or controlled experiment. It's the critical step between forming a hypothesis and launching the test. It answers the question, 'How many users do we need to expose to this test to confidently measure the outcome?' This prevents you from running statistically underpowered experiments that waste traffic and lead to false negatives.

WHEN NOT TO USE IT MDE is a planning tool, not an analysis tool. After an experiment is complete, you should report the observed effect, confidence intervals, and p-value, not the MDE you planned for. MDE is also less relevant for purely exploratory data analysis where you have no specific prior hypothesis to test.

ONE CANONICAL EXAMPLE A product team wants to test a new signup button. The current conversion rate is 5%. They decide that any improvement less than 0.5% isn't worth the engineering effort. Their MDE is therefore 0.5%. Using a power calculator, they find that to reliably detect a 0.5% lift (from 5% to 5.5%) with 80% power, they need 25,000 users in both the control and variant groups. If they can't get that much traffic in a reasonable time, they must either accept a higher MDE (e.g., 1%) or scrap the test.