Explain Simpson's Paradox and construct a user engagement scenario

WHAT THIS TESTS: Your ability to recognize that aggregated statistics can reverse the direction of subgroup trends when the composition of the groups differs. Interviewers care whether you understand confounding by population mix, can construct a coherent numerical example, and know how this paradox breaks A/B tests and product decisions.

A GOOD ANSWER COVERS: First, a crisp definition of Simpson's Paradox as a statistical phenomenon where a trend appears in several groups but disappears or reverses when the groups are combined. Second, a concrete hypothetical scenario with two segments such as new and returning users. Third, real numbers showing the treatment wins in every segment but loses in aggregate because the treatment group contains a higher proportion of low-engagement users. Fourth, the root cause explanation that the aggregate is a weighted average skewed by unequal segment sizes or traffic allocation, which masks the true directional effect. Fifth, a mitigation strategy such as maintaining consistent traffic splits, checking for confounders, and using segment analysis only for discovery rather than shipping decisions.

COMMON WRONG ANSWERS: Blaming the paradox on bad data or noisy metrics without naming the skewed population mix. Offering a scenario where the feature wins in one segment and loses in another but still loses overall; that is not Simpson's Paradox because the paradox requires the same directional effect in every subgroup. Suggesting that segment-level results should override the aggregate and justify shipping the feature. Failing to mention that inconsistent traffic splits across weeks or cohorts can manufacture the paradox in an experiment.

LIKELY FOLLOW-UPS: How would you detect this in a live A/B test? What is the difference between a confounder and an interaction effect? If the aggregate is negative but every segment is positive, should you ship the feature? How do you prevent this during experiment design?

ONE CONCRETE EXAMPLE: Suppose a targeting bug places 80 percent new users and 20 percent returning users into treatment, while control gets the inverse. New users baseline at 10 percent engagement and returning users at 50 percent. With the feature, new users rise to 12 percent and returning users rise to 55 percent, so the feature wins in every segment. Yet the treatment blend is roughly 20.6 percent while the control blend is 42 percent, producing an aggregate drop even though every subgroup improved.

Source: statsig.com