How would you validate a feature's conversion impact given self-selection bias?

What's really being asked

Your ability to recognize that adoption of an optional feature is not randomly assigned. Users who opt in likely differ systematically from those who do not in ways that also affect conversion, such as intent, budget, or sophistication. The interviewer wants to see if you can design an analysis that moves beyond correlation to estimate a causal treatment effect using observational or quasi-experimental methods.

The full answer

Four elements in order. First, explicitly state the self-selection problem: adopters are a non-random subset with higher baseline propensity to convert, so a raw comparison overstates the feature's true impact. Second, propose a quasi-experimental design to construct a credible counterfactual. Options include propensity score matching or coarsened exact matching to pair adopters with statistically similar non-adopters based on pre-adoption covariates; difference-in-differences if you have time-series data and can verify parallel pre-trend trajectories; or an instrumental variable such as a subtle UI nudge or eligibility rule that affects adoption but not conversion directly. Third, describe falsification and robustness checks, such as testing for pre-trend parity, placebo tests on fake adoption dates, or sensitivity analysis for unobserved confounders. Fourth, translate the statistical result into a business decision by estimating the incremental conversion lift and weighing it against the engineering cost of making the feature default or promoting it more broadly.

The mistakes people make

Proposing a simple t-test or chi-squared test on conversion rates between adopters and non-adopters without any correction for selection bias. Running a logistic regression with adoption as a predictor while omitting key confounders that drove both adoption and conversion. Confusing this with an A/B test scenario where randomization already occurred. Claiming that controlling for a few demographic variables is sufficient without discussing the parallel trends assumption or unobserved heterogeneity.

What usually comes next

How would you handle unobserved confounders if matching does not fully balance the groups? What would you do if the feature adoption is rare, making it hard to find good matches? How would you design a randomized experiment to get a ground-truth estimate? What if adoption happens gradually over time and users can switch in and out?

A concrete example

Suppose 5 percent of users adopt a new advanced search filter and convert at 20 percent versus 8 percent for non-adopters. Instead of declaring a 12 point lift, you match each adopter with three non-adopters who had identical pre-adoption behavior, account age, and past search counts. After matching, the gap shrinks to 3 percentage points. You then run a difference-in-differences model on weekly conversion and confirm parallel pre-trends, yielding a credible causal estimate of a 2.5 point lift. You conclude the feature helps, but not as dramatically as the raw numbers suggest, and recommend an opt-out trial to verify.