Propensity Score Matching: Mimicking an A/B Test

The mental model

Propensity Score Matching (PSM) creates a fair comparison group after the fact, when you couldn't run a proper A/B test. Imagine you launched a new feature that users could opt into. Are the adopters happier because of the feature, or were they already your most engaged users? PSM isolates the feature's effect by finding non-adopters who looked almost identical to the adopters before the feature was available.

How it works

PSM is a two-step process. First, you build a statistical model (often logistic regression) to calculate a "propensity score" for every user. This score is the predicted probability that a user would receive the "treatment" (e.g., adopt the feature) based on their pre-treatment characteristics, or covariates, like activity level, account age, and location. Second, for each treated user, you find one or more untreated users with a very similar propensity score. This matched set of untreated users becomes your new, cleaner control group. You can then compare outcomes between the treated group and their statistical twins.

When to use it

Use PSM for causal inference on observational data where randomization isn't feasible. This is common when measuring the impact of a user-activated feature, a non-random marketing campaign, or a policy change. It's your best bet when a true A/B test is impractical, unethical, or was simply never run.

When not to use it

Never choose PSM over a well-designed randomized controlled trial (RCT), which is always the gold standard for causality. PSM is also ineffective if there is no "common support"—meaning the treated and untreated groups are so different that you can't find good matches. Most importantly, PSM cannot control for unobserved confounders. If a key factor influencing treatment selection (like user intent) isn't in your data, your results will still be biased.

One canonical example

A company wants to know if its new, optional premium subscription increases 90-day retention. A simple comparison is biased because subscribers are likely more engaged to begin with. Using PSM, they first model the propensity to subscribe based on pre-subscription data like daily sessions and features used. Then, for each subscriber, they find a non-subscriber with a nearly identical propensity score. By comparing the retention rates of these two matched groups, they get a much cleaner estimate of the subscription's true effect.