How would you validate a feature's success, correcting for self-selection?

WHAT THIS TESTS: This question tests your ability to recognize and correct for self-selection bias, a common confounder in observational data. The interviewer is looking for statistical maturity beyond simple A/B testing. They want to see if you can move from identifying a correlation (adopters convert more) to designing an analysis that estimates a causal link (the feature causes more conversion). It's a test of analytical rigor and understanding the limits of non-randomized data.

A GOOD ANSWER COVERS: A strong answer has four parts. First, explicitly state the problem: the group that opted into the feature is not random. They are likely more engaged or motivated, so comparing them to the general population is misleading. Second, propose creating a synthetic control group using a quasi-experimental method, since a true A/B test was not run. Third, detail a specific technique like Propensity Score Matching (PSM). Explain that you would model the probability of a user adopting the feature based on their characteristics before the feature launched (e.g., login frequency, past purchases). Then, for each adopter, find one or more non-adopters with a very similar propensity score to create a matched control group. Fourth, compare the conversion rates between the treatment group (adopters) and the matched control group to get a more accurate estimate of the feature's impact. Mentioning Difference-in-Differences (DiD) as an alternative is also a strong signal.

COMMON WRONG ANSWERS: A major red flag is accepting the PM's conclusion or suggesting a simple analysis that doesn't control for the bias. For example, just segmenting users by 'active' vs. 'inactive' is too crude and doesn't solve the core self-selection problem. Another weak answer is to only point out the bias without offering a concrete, technical solution for how to address it. Finally, suggesting an A/B test now without a plan for how to handle users who have already self-selected is an incomplete thought. The key is to work with the observational data you have.

LIKELY FOLLOW-UPS: "What if you don't have good pre-feature data to build a propensity score model?" (Answer: Acknowledge the limitation, state the analysis will be less reliable, and suggest other methods like Difference-in-Differences if you have time-series data). Another follow-up: "How would you explain the results of your more complex analysis to the product manager?" (Answer: Focus on the intuition. "We found a group of users who looked identical to the adopters right before the feature launched, but didn't use it. Their conversion rate was almost as high, which means the feature itself isn't the main driver of success.")

ONE CONCRETE EXAMPLE: Imagine the PM reports that adopters have a 15% conversion rate, while non-adopters have a 3% rate. This is a 12 percentage point difference. After running a Propensity Score Matching analysis, you find a matched control group of non-adopters who had similar pre-feature engagement levels. This matched group has a 13% conversion rate. Your conclusion is that the feature's true causal lift is only 2 percentage points (15% - 13%), not 12. The other 10 points were due to self-selection—the most engaged users would have converted at a high rate anyway.