Stakeholder claims correlation implies causation. How do you investigate?

WHAT THIS TESTS: This question tests your data literacy and scientific rigor. The interviewer wants to see if you can move beyond the cliché "correlation does not imply causation" and apply a structured, skeptical approach. They are evaluating your ability to diplomatically challenge a stakeholder, design follow-up analyses, and ultimately guide the team toward a decision based on evidence, not just association. It's a test of your ability to think like a scientist in a business context.

A GOOD ANSWER COVERS: A strong answer has four parts. First, acknowledge and validate the stakeholder's observation as an interesting starting point, showing you're a collaborative partner. Second, introduce the concept of confounding variables. Ask what other factors might be at play. For example, are the users of feature A simply more engaged "power users" who use many features and would have high retention anyway? Third, propose specific, non-experimental analyses to dig deeper. This includes cohort analysis (do users who start using feature A have better retention than those who don't?) and segmentation (does this hold for new vs. tenured users, or across different geographies?). Fourth, propose the gold standard for proving causality: a controlled experiment (A/B test). Suggest running a test where you proactively expose a random segment of new users to feature A and measure their retention against a control group that isn't exposed.

COMMON WRONG ANSWERS: The biggest red flag is simply stating "correlation doesn't imply causation" (or the Latin cum hoc ergo propter hoc) and stopping there. This is correct but unhelpful; it sounds academic and dismissive, not collaborative. Another mistake is immediately jumping to an A/B test without first suggesting cheaper, faster analyses like segmentation, which shows a lack of pragmatism. A third error is accepting the conclusion at face value without any critical thought, which signals a lack of analytical depth.

LIKELY FOLLOW-UPS: Expect questions like "An A/B test is too expensive/slow. What's the next best thing we can do to increase our confidence?" (Answer: Causal inference methods like propensity score matching or regression discontinuity). Another follow-up could be "How would you design that A/B test? What are the primary and secondary metrics?" (Answer: Primary metric is N-day retention; secondary metrics could be session length, usage of other features, etc.).

ONE CONCRETE EXAMPLE: Let's say feature A is "profile completion." Users with complete profiles have 20% higher 30-day retention. Instead of pushing everyone to complete their profile, first check if these users are just your most motivated cohort. A better step is to run an experiment: for 50% of new users, show them an aggressive "complete your profile" onboarding flow (the variant). The other 50% get the standard flow (the control). If the variant group shows a statistically significant lift in 30-day retention, you have evidence of causation. If not, the original correlation was spurious.