Average Treatment Effect (ATE): Isolating the Impact of a Change

The mental model

The Average Treatment Effect (ATE) answers the question, "Did this change actually work?" Imagine two parallel universes: in one, you ship a new feature, and in the other, you don't. The ATE is the difference in outcomes between them. Since we can't access parallel universes, we use randomized treatment and control groups as our best approximation to isolate the change's causal impact.

How it works

In a randomized experiment, the ATE is estimated by a simple subtraction: (Mean outcome of the treatment group) - (Mean outcome of the control group). By randomly assigning units (like users or patients) to each group, we ensure that, on average, the only systematic difference between them is the treatment itself. Any resulting difference in their average outcomes can therefore be attributed to the treatment's effect.

When to use it

ATE is the core metric for establishing causality in controlled experiments. Three common places you'll see it are: first, in software A/B tests to measure if a new button color increases clicks; second, in policy evaluations to see if a job program raised income levels; and third, in medical trials to determine if a new drug improved patient outcomes compared to a placebo.

When not to use it

Avoid calculating a simple ATE on observational data where groups self-select. For example, comparing the income of people who chose to get a certification versus those who didn't. The groups aren't random; the people seeking certification might be more ambitious to begin with. This 'selection bias' makes it impossible to claim the certification caused the income difference without using more advanced statistical methods to control for confounding variables.

One canonical example

An e-commerce site A/B tests a new, simplified checkout flow. The treatment group (10,000 users) sees the new flow and has a purchase conversion rate of 4%. The control group (10,000 users) sees the old flow and has a conversion rate of 3%. The estimated ATE is 4% - 3% = +1 percentage point. This suggests the new checkout flow caused a 1 percentage point increase in conversions.