Use Difference-in-Differences without an A/B test
causal inference when randomization is impossible.
give a scenario like a region-wide launch, apply Difference-in-Differences comparing treated vs control over time, and state the parallel-trends assumption.
What's really being asked
The interviewer wants to know whether you can still make a credible causal claim when a clean randomized experiment is off the table, and whether you understand the assumption that makes the alternative valid.
The full answer
First, a realistic scenario where randomization fails: a feature that must launch to everyone at once for legal, contractual, or network-effect reasons; a brand or pricing change that cannot be split by user; or a regional rollout mandated by regulation. In such cases, propose Difference-in-Differences. You pick a treated group that received the change and a comparable control group that did not, measure the outcome before and after for both, and estimate the effect as the difference in the treated group's change minus the difference in the control group's change. This cancels out any trend or shock that hit both groups equally. State the core assumption explicitly: parallel trends, meaning that in the absence of the launch, the treated and control groups would have continued moving in parallel. You support it by showing the two groups tracked together in the pre-period.
The mistakes people make
Comparing only before and after for the treated group with no control, which confounds the launch with seasonality, marketing, or macro shifts. Choosing a control group that was already trending differently, violating parallel trends. Claiming DiD proves causation regardless of its assumption.
What usually comes next
How do you test the parallel-trends assumption? What other quasi-experimental methods exist, such as synthetic control, regression discontinuity, or instrumental variables? What threats like spillover between groups could bias the estimate?
A concrete example
You launch a new pricing page only in Canada due to a regulatory requirement, while the US keeps the old page. Conversion rises 5 points in Canada and 2 points in the US over the same window. Difference-in-Differences attributes a 3-point lift to the launch, assuming that without it Canada would have risen by the same 2 points as the US.
Interview question
What is the core assumption that must hold for a Difference-in-Differences estimate to be valid?
- a.The treatment must be randomly assigned to individuals
- b.Absent the treatment, the treated and control groups would have followed parallel trendsCorrect
- c.The treated and control groups must be exactly equal in size
- d.The outcome must be normally distributed in both groups
Why? this is the answer
DiD relies on parallel trends: without the intervention the two groups would have moved together, so any divergence is attributable to the treatment. Random assignment is exactly what DiD substitutes for, and equal sizes or normality are not required.
Just read this? Test yourself on what you have been reading.
Read the original → statsig.com
- #causal-inference
- #difference-in-differences
- #quasi-experiment
- #parallel-trends
- #experimentation
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles