Stakeholder claims feature usage drives retention. How do you validate?

What's really being asked

This question tests whether you can spot the cum hoc ergo propter hoc fallacy in a business context and move beyond dashboard correlations to causal reasoning. The interviewer wants to see that you treat retention as an outcome requiring a counterfactual, not as a variable that can be manipulated by shipping more of a feature without evidence.

The full answer

A strong response starts with three critical questions. First, what is the direction of causality: does feature usage drive retention, or do retained users simply have more sessions in which to encounter the feature? Second, what confounders explain both variables, such as user segment, onboarding depth, or account age? Third, is there selection bias because power users are more likely to discover advanced features? After raising these questions, the candidate should propose validation methods. These include natural experiments like geographic or platform holdouts, difference-in-differences around a launch, instrumental variables such as exposure to a tooltip or notification, and propensity score matching to compare similar users who did and did not adopt the feature. For visualizations, a great answer suggests plotting retention curves for propensity-matched cohorts, lagging usage by one or two weeks to see if adoption precedes retention lifts, and examining funnel drop-off before the feature to rule out reverse causality.

The mistakes people make

The biggest red flag is agreeing with the stakeholder and proposing immediate investment without questioning the causal mechanism. Another weak pattern is listing generic data quality issues like missing events or small sample size without addressing the core logic of causation. A subtler mistake is suggesting an A/B test that randomizes feature access without considering that forcing usage may create artificial behavior that does not match organic adoption.

What usually comes next

The interviewer may push you to design the exact A/B test or quasi-experiment, asking how you would handle network effects if the feature is social. They might also ask how you would quantify the opportunity cost of investing in feature A versus other retention levers, or how you would communicate uncertainty to a stakeholder who has already made up their mind.

A concrete example

Imagine a collaboration tool where users who create shared workspaces show 40 percent higher six-month retention. Before investing in workspace templates, you check whether users who were invited to a workspace, rather than creating one organically, also retain better. You run a propensity-matched analysis and find that invitees retain at the same rate as non-users, suggesting the correlation is driven by pre-existing team adoption rather than the feature itself. You recommend a holdout experiment that shows workspace creation prompts to only half of eligible users and measures lagged retention.