Causal Impact: Measuring Effects Without an A/B Test

WHY IT EXISTS You launched a big ad campaign. Did it work? You can't run a true A/B test because everyone saw the ads. You need a way to estimate the effect after the fact, separating the campaign's impact from normal fluctuations. Causal Impact was designed for this "no control group" scenario.

THE MENTAL MODEL Imagine two parallel universes. In one, you launched the ad campaign. In the other, you didn't. Causal Impact uses data from before the launch to build a statistical model of the "you did nothing" universe. It then compares the actual results from our universe to its prediction, and the difference is the estimated causal effect.

HOW IT WORKS The technique uses a Bayesian structural time-series model. It takes your main metric (e.g., daily clicks) and one or more "control" time series that are correlated with your main metric but were not affected by the intervention (e.g., clicks on an unrelated product). The model learns the relationship between these series before the intervention. After the intervention date, it uses the control series to predict what the main metric would have been. The difference between this prediction and the actual observed metric is the "causal impact".

WHEN TO USE IT Use it when a randomized controlled experiment (like an A/B test) is impossible, impractical, or was simply not done. It's ideal for analyzing the effect of events that affect an entire population, such as a new law, a marketing campaign, a site-wide redesign, or a major news event.

WHEN NOT TO USE IT Do not use it if you can run a proper A/B test; a randomized experiment is always superior. The method fails if you can't find control time series that are both correlated with your outcome and truly unaffected by the intervention. If the underlying relationship between your metrics changes for reasons other than the intervention, the model's assumptions are violated and its output is unreliable.

ONE CANONICAL EXAMPLE An advertiser wants to know how many extra clicks an ad campaign generated. They use Causal Impact, feeding it the daily clicks for the advertised product (the treated series) and daily clicks for several other, un-advertised products (the control series). The model uses the pre-campaign period to learn how these series move together, then projects the expected clicks post-campaign. The difference between the actual clicks and the projected clicks is the estimated lift from the campaign.