When is A/B testing not feasible, and what is an alternative?
Tests your grasp of causal inference when randomization isn't possible. A great answer names a scenario (like a regional launch), proposes Difference-in-Differences (DiD), and explains its core 'parallel trends' assumption.
What's really being asked
This question tests your depth in analytics and causal inference. Interviewers want to see if you can think beyond standard A/B tests and still measure impact rigorously. It separates senior candidates, who have a toolkit for messy, real-world scenarios, from junior candidates who may only know how to analyze a clean experiment. It's a test of practical data science and product sense.
The full answer
An ideal answer has four parts. First, identify a clear scenario where user-level randomization is impossible or impractical, such as a feature launch constrained by geography (e.g., a new service in California only), a hardware update, or a marketplace policy change that affects everyone. Second, propose Difference-in-Differences (DiD) as the appropriate quasi-experimental method. Third, explain how DiD works: it calculates the change in a metric over time for the treatment group and subtracts the change in the same metric over the same period for the control group. This double-difference isolates the treatment effect from general trends. Fourth, and most critically, state the core assumption of DiD: the parallel trends assumption. This means that, absent the intervention, the treatment and control groups would have followed similar trends over time.
The mistakes people make
One major red flag is proposing a simple pre-vs-post analysis on the treated group, which fails to control for any external factors or seasonality. Another common mistake is misstating the parallel trends assumption, for example, by saying the groups must have identical metric values before the test (they only need to trend similarly). A weaker answer might name DiD but be unable to explain the assumption, suggesting rote memorization. Finally, failing to propose a control group at all is a fundamental error.
What usually comes next
Be ready for: "How would you test the parallel trends assumption?" (The answer is to plot the metric for both groups in the pre-treatment period and visually inspect if the trends are parallel). Another follow-up could be, "What happens if the trends are not parallel?" (You could mention looking for a better control group or using more advanced methods like synthetic controls). A third question might be, "What other quasi-experimental methods are you familiar with?" (e.g., Interrupted Time Series, Regression Discontinuity).
A concrete example
Imagine we launch a new feature for all users in Canada, but not the US. An A/B test is impossible. We can use DiD. The treatment group is Canadian users; the control group is US users. We measure daily active users (DAU) for 4 weeks before the launch (pre-period) and 4 weeks after (post-period). Let's say pre-launch, Canada DAU was 100k and US DAU was 1M. Post-launch, Canada DAU is 125k and US DAU is 1.1M. The simple growth in Canada is +25k. But the US also grew by 100k (a 10% increase), suggesting a market trend. The DiD estimate is (125k - 100k) - (1.1M - 1.0M) = 25k - 100k. This is wrong. You must use rates. The US grew 10%. The expected growth for Canada was 10% of 100k, so 10k. The actual growth was 25k. The treatment effect is 25k - 10k = 15k increase in DAU.
Interview question
When using Difference-in-Differences (DiD) to estimate a causal effect in a quasi-experiment, which assumption is most critical for its validity?
- a.The treatment effect must be constant for all units in the treatment group.
- b.The intervention must be applied randomly to individuals within the treatment group.
- c.Absent the intervention, the treatment and control groups would have followed similar trends over time.Correct
- d.The treatment and control groups must have identical average metric values before the intervention.
Why? this is the answer
The core assumption of DiD is the parallel trends assumption, meaning that without the intervention, both groups would have evolved similarly. Option D is a common misconception; the groups do not need identical starting values, only similar trends.
Just read this? Test yourself on what you have been reading.
Read the original → statsig.com
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on causal inference — each one lists the topics its interview covers.
See open roles