How would you estimate causal impact using a quasi-experimental method?
Causal inference via difference-in-differences.
DiD with Canada versus Australia; assert parallel trends; validate with pre-period plots and placebo tests.
What's really being asked
Your ability to identify and defend a quasi-experimental strategy when true randomization is unavailable. The interviewer wants to see if you understand that simple cross-sectional comparisons are biased and that you can articulate the identifying assumptions behind difference-in-differences, the standard tool for this scenario.
The full answer
First, name difference-in-differences as the primary method and set up the two-by-two structure: Canada is the treatment group, Australia is the control, and you compare the before-after change in Canada to the before-after change in Australia. Second, explicitly state the parallel trends assumption, which is that in the absence of the feature both countries would have experienced the same evolution in engagement metrics. Third, describe how to test this assumption using pre-treatment data, specifically by running an event study or plotting trends to verify no divergence before launch. Fourth, mention placebo tests such as assigning a fake launch date to Australia to confirm no spurious effect. Fifth, discuss robustness alternatives like synthetic control if parallel trends is questionable.
The mistakes people make
A major red flag is proposing a simple post-launch comparison of means between Canada and Australia without any pre-period adjustment. Another weak answer is confusing correlation with causation by attributing all Canadian engagement changes to the feature while ignoring seasonality or macro trends. Some candidates mention A/B testing, which is operationally impossible here since the feature was launched nationally. Failing to name parallel trends or offering no way to validate it also signals shallow familiarity with causal inference.
What usually comes next
The interviewer may ask what you would do if pre-trends diverge, in which case you should discuss synthetic control methods or finding a better control group. They might probe heterogeneous treatment effects, asking how the impact varies by user segment, which leads to interacting the DiD estimator with cohort indicators. Another follow-up is handling staggered rollout across multiple countries, which complicates the standard two-period DiD and requires recent advances in staggered difference-in-differences estimators.
A concrete example
Suppose daily active user rate in Canada was ten percent before launch and twelve percent after, while Australia moved from nine percent to nine point five percent over the same period. The DiD estimate is the difference of differences, which is one point five percentage points. You would then run a placebo test by pretending the feature launched three months earlier and checking that the coefficient is near zero, and you would plot weekly engagement for both countries for six months pre-launch to visually confirm parallel trends.
Interview question
Using difference-in-differences with Canada as treatment and Australia as control, what is the most direct way to validate the parallel trends assumption?
- a.Plotting pre-launch engagement trends for both countries to confirm they did not divergeCorrect
- b.Running a placebo test that assigns a fake launch date to Australia to confirm no spurious effect
- c.Checking that both countries had identical engagement levels immediately before the launch date
- d.Comparing post-launch engagement means between the two countries while ignoring any pre-period data
Why? this is the answer
Examining pre-launch trends directly tests whether the control group is a valid counterfactual by verifying both groups would have evolved similarly absent treatment. A placebo test is a valuable robustness check, but it does not directly assess whether the parallel trends assumption holds; it only checks for spurious effects under a fake treatment date.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #causal inference
- #difference in differences
- #quasi-experiment
- #product analytics
- #econometrics
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on causal inference — each one lists the topics its interview covers.
See open roles