Is 20% higher retention from Feature X causal or correlational?

WHAT THIS TESTS: This question tests your scientific rigor and business sense. The interviewer is checking if you can recognize that correlation is not causation and if you know the standard industry tool (A/B testing) to prove a hypothesis before committing more resources. It's about avoiding common, expensive product fallacies.

A GOOD ANSWER COVERS: First, immediately question the premise by identifying potential confounding variables. The users who discovered and used Feature X are likely not average users; they might be power users or more engaged in general, which is the real cause of their higher retention.

Second, propose a randomized controlled trial (an A/B test) as the gold standard for determining causality. This demonstrates you know how to get a reliable answer.

Third, outline the specifics of the experiment. This includes defining the control group (users who do not see the feature), the treatment group (users who do), the primary metric (e.g., 30-day retention), and the population (e.g., new users, to avoid selection bias). Mentioning guardrail metrics (e.g., engagement with other features) is a plus.

Fourth, define what success looks like. For example, if the treatment group shows a statistically significant retention lift (e.g., a 2% absolute lift with a p-value < 0.05), then you can conclude the feature has a causal impact.

COMMON WRONG ANSWERS: Jumping to conclusions. A huge red flag is saying, "A 20% lift is great, we should invest more in this feature." This shows a lack of analytical discipline and an inability to see beyond the surface-level data.

Proposing only a weaker, retrospective analysis. While methods like propensity score matching are valid, they are not the first-choice answer when a direct experiment is possible. Leading with this suggests you may not have experience running A/B tests.

Giving a vague answer like "we should run an A/B test" without specifying the experimental design. The details (groups, metrics, population) are what separate a junior answer from a senior one.

LIKELY FOLLOW-UPS: "What if you can't run an A/B test for technical or ethical reasons?" (This is a prompt to discuss quasi-experimental methods like difference-in-differences or regression discontinuity). "How would you determine the required sample size and duration for this test?" (Tests knowledge of statistical power analysis). "The test result is flat. What's your recommendation?" (Tests product sense and ability to interpret null results).

ONE CONCRETE EXAMPLE: A social media app finds that users who use the 'Stories' feature have 20% higher retention. The confounding factor is that users who create content are already more engaged. To test causality, you would run an A/B test on 100,000 new users. 50,000 (control) get the app without the Stories feature prominent. 50,000 (treatment) get the normal app. If the treatment group's Day-14 retention is 35% and the control's is 32% (a statistically significant difference), you can attribute the 3% lift to the feature.