Sequential Testing: Stop When You Have Enough Data

Why it exists

Traditional A/B testing requires you to calculate a sample size and run the test to completion, even if a clear winner emerges early. This is often inefficient, wasting traffic, time, and money. Sequential testing was developed to solve this by allowing for early stopping without compromising statistical validity, answering the question: "Can I stop this test now?".

The mental model

Think of it like a race. In a classic fixed-horizon test, all runners must complete the full 100-meter dash. In a sequential test, the race ends as soon as one runner is so far ahead that their victory is statistically inevitable. You don't make them run the full distance just for the sake of it. The key is defining "far enough ahead" with statistical rigor before the race starts.

How it works

Instead of one analysis at the end, sequential testing involves continuous or periodic monitoring. At each check-in, you apply a special statistical test that accounts for the fact you're looking multiple times. These tests use "stopping boundaries." If your test statistic (like the difference in conversion rates) crosses a boundary, you stop the test and declare a winner or conclude there's no difference. These boundaries are wider than in a fixed-horizon test to control the rate of false positives that arises from repeated peeking.

When to use it

Use sequential methods when the cost of experimentation is high or the opportunity cost of delaying a decision is significant. It's ideal for high-traffic A/B testing platforms, clinical trials where patient outcomes are at stake, or any online experiment where you want to iterate faster. If you want to ship a winning feature the moment it proves its worth, this is the tool.

When not to use it

Avoid sequential testing if you need to measure the precise magnitude of an effect, not just declare a winner. Because they are designed to stop early, sequential tests can overestimate the effect size of the winning variant (an effect known as the "winner's curse"). They are also more complex to set up than traditional tests. If you lack the right statistical tools, a simple fixed-horizon test is safer.

One canonical example

An e-commerce site tests a new checkout button. Using a sequential approach, they monitor conversions daily with a stopping rule: "Stop if the new button is winning with 95% significance, or after 4 weeks." On day 5, the new button's performance crosses the significance boundary. They stop the test and ship the change, saving weeks of running the experiment and capturing the revenue lift immediately.