Statistical Power: Detecting Real Effects in Your Tests

THE MENTAL MODEL: Statistical power is the probability that your test will detect an effect that actually exists. Imagine your experiment is a smoke detector. Power is the chance the alarm will sound during a real fire. A test with 80% power has an 80% chance of finding a real effect but a 20% chance of a 'false negative' — missing it completely.

HOW IT WORKS: Power is a function of three main levers. First, effect size: it's easier to detect a large, obvious change than a small, subtle one. Second, sample size: more data reduces noise and increases the chance of seeing the true signal. This is the most common variable you adjust. Third, the specific statistical test used, as different tests have different sensitivities.

WHEN TO USE IT: Power analysis is a planning tool used before you run an experiment. It answers the question: "How many users do I need to be reasonably sure of detecting a meaningful change?" This is standard practice for designing A/B tests, scientific studies, and clinical trials to avoid wasting resources on tests that are doomed to be inconclusive.

WHEN NOT TO USE IT: Power is a concept from frequentist statistics. It is not a primary concern in other frameworks like Bayesian analysis, which focuses on updating beliefs based on evidence. Critically, you do not calculate power after an experiment is over. This 'post-hoc' power analysis is misleading; the test either found a significant result or it didn't.

ONE CANONICAL EXAMPLE: A product team wants to test a new signup button. They only care about changes greater than a 2% lift (effect size) and want an 80% chance of detecting such a lift if it exists (power). A power calculation tells them they need a sample size of 25,000 users per variant. Without this, they might have tested on only 2,000 users, found no effect due to low power, and wrongly concluded the new button was a failure.