Use Difference-in-Differences without an A/B test

WHAT THIS TESTS The interviewer wants to know whether you can still make a credible causal claim when a clean randomized experiment is off the table, and whether you understand the assumption that makes the alternative valid.

A GOOD ANSWER COVERS First, a realistic scenario where randomization fails: a feature that must launch to everyone at once for legal, contractual, or network-effect reasons; a brand or pricing change that cannot be split by user; or a regional rollout mandated by regulation. In such cases, propose Difference-in-Differences. You pick a treated group that received the change and a comparable control group that did not, measure the outcome before and after for both, and estimate the effect as the difference in the treated group's change minus the difference in the control group's change. This cancels out any trend or shock that hit both groups equally. State the core assumption explicitly: parallel trends, meaning that in the absence of the launch, the treated and control groups would have continued moving in parallel. You support it by showing the two groups tracked together in the pre-period.

COMMON WRONG ANSWERS Comparing only before and after for the treated group with no control, which confounds the launch with seasonality, marketing, or macro shifts. Choosing a control group that was already trending differently, violating parallel trends. Claiming DiD proves causation regardless of its assumption.

LIKELY FOLLOW-UPS How do you test the parallel-trends assumption? What other quasi-experimental methods exist, such as synthetic control, regression discontinuity, or instrumental variables? What threats like spillover between groups could bias the estimate?

ONE CONCRETE EXAMPLE You launch a new pricing page only in Canada due to a regulatory requirement, while the US keeps the old page. Conversion rises 5 points in Canada and 2 points in the US over the same window. Difference-in-Differences attributes a 3-point lift to the launch, assuming that without it Canada would have risen by the same 2 points as the US.