Isolating Impact with Difference-in-Differences (DiD)

THE MENTAL MODEL: Difference-in-Differences (DiD) is like running a controlled experiment when you can't actually randomize who gets the treatment. Instead, you find a naturally occurring control group and measure how much more the treatment group changed compared to the control group. It answers the question: "After accounting for background trends that affect everyone, what was the additional, isolated effect of our intervention?"

HOW IT WORKS: You need four key numbers: the average outcome for the treatment group before and after the change, and the same two numbers for the control group. The DiD effect is calculated as (Treatment_After - Treatment_Before) - (Control_After - Control_Before). This double subtraction first finds the change within each group, then finds the difference between those two changes. This isolates the treatment effect from any general trend, like seasonality or market growth, that would have affected both groups.

WHEN TO USE IT: Use DiD when you have observational data from before and after an intervention for both a group that received it (treatment) and a similar group that did not (control). This is common for analyzing policy changes (e.g., a new law in one state but not another), feature rollouts to a subset of users, or marketing campaigns in specific regions. It is a powerful tool for causal inference without a true randomized controlled trial.

WHEN NOT TO USE IT: The method's biggest footgun is violating the "parallel trends assumption." Do not use DiD if the treatment and control groups were already on different trajectories before the intervention. For example, comparing your most active users (treatment) to your least active users (control) is invalid because their trends were never parallel. Also, avoid it if other significant events happen at the same time as the treatment that affect one group but not the other, as this confounds the results.

ONE CANONICAL EXAMPLE: You launch a new feature for users in California (treatment) but not in Texas (control). Before launch, CA users averaged 10 key actions/week and TX users averaged 8. After launch, CA users average 15 actions/week and TX users average 9. The simple change in CA is +5. But the background trend, seen in TX, is +1. The DiD effect is (15 - 10) - (9 - 8) = 5 - 1 = 4. The feature's true impact was likely a 4-action increase per week.