How would you validate a PM's claim about feature success?

WHAT THIS TESTS: This question tests your ability to distinguish correlation from causation and your understanding of analytical techniques beyond a simple A/B test. The interviewer is looking for your grasp of causal inference and how to handle common biases, like the self-selection bias described in the prompt. They want to see if you can design a rigorous post-hoc analysis when a clean experiment was not run.

A GOOD ANSWER COVERS: A strong answer has four parts. First, acknowledge the PM's data but immediately identify the self-selection bias as a major confounder; users who opt-in are likely more engaged to begin with. Second, state that since a true A/B test is no longer possible, you need to use a quasi-experimental method to approximate a control group. Third, propose a specific technique like Propensity Score Matching (PSM). For PSM, you would model the probability of a user adopting the feature based on their characteristics before the feature was launched (e.g., login frequency, tenure, past conversion). Fourth, explain that you would then compare the conversion rates between the feature adopters and a matched control group of non-adopters who had a very similar propensity score. This isolates the feature's effect more accurately.

COMMON WRONG ANSWERS: A major red flag is taking the PM's initial analysis at face value and agreeing the feature is a success. Another is suggesting an impossible solution, like "we should have run an A/B test," without offering a way to analyze the existing data. A vague answer like "we need to control for other factors" is also weak; a senior candidate is expected to name the statistical method they would use (e.g., matching, regression adjustment) and explain how it works. Finally, suggesting you can fix this by just running an A/B test on new users misses the point of evaluating the impact on the current user base.

LIKELY FOLLOW-UPS: Be prepared for "What if you don't have good pre-feature data to build a propensity model?" (Answer: The analysis becomes much less reliable; you might have to use cruder matching on demographics, or state that a causal claim is impossible). Or, "What's another method besides PSM?" (Answer: Difference-in-Differences, Instrumental Variables, or Regression Discontinuity if the feature was rolled out based on a threshold).

ONE CONCRETE EXAMPLE: The PM sees that adopters convert at 6% while non-adopters convert at 2%, a 200% lift. You suspect the adopters were already power users. You build a propensity model and find that the most similar non-adopters (your matched control group) actually had a baseline conversion rate of 5%. The comparison is now 6% vs 5%, a 20% relative lift. This is a much more realistic and defensible estimate of the feature's true impact.