Explain Simpson's Paradox and construct a user engagement scenario
Tests whether you spot trends reversing when population mixes differ. Good answers define the paradox, give a numerical example with per-segment wins but aggregate loss, and warn against segment-only decisions.
What's really being asked
Your ability to recognize that aggregated statistics can reverse the direction of subgroup trends when the composition of the groups differs. Interviewers care whether you understand confounding by population mix, can construct a coherent numerical example, and know how this paradox breaks A/B tests and product decisions.
The full answer
First, a crisp definition of Simpson's Paradox as a statistical phenomenon where a trend appears in several groups but disappears or reverses when the groups are combined. Second, a concrete hypothetical scenario with two segments such as new and returning users. Third, real numbers showing the treatment wins in every segment but loses in aggregate because the treatment group contains a higher proportion of low-engagement users. Fourth, the root cause explanation that the aggregate is a weighted average skewed by unequal segment sizes or traffic allocation, which masks the true directional effect. Fifth, a mitigation strategy such as maintaining consistent traffic splits, checking for confounders, and using segment analysis only for discovery rather than shipping decisions.
The mistakes people make
Blaming the paradox on bad data or noisy metrics without naming the skewed population mix. Offering a scenario where the feature wins in one segment and loses in another but still loses overall; that is not Simpson's Paradox because the paradox requires the same directional effect in every subgroup. Suggesting that segment-level results should override the aggregate and justify shipping the feature. Failing to mention that inconsistent traffic splits across weeks or cohorts can manufacture the paradox in an experiment.
What usually comes next
How would you detect this in a live A/B test? What is the difference between a confounder and an interaction effect? If the aggregate is negative but every segment is positive, should you ship the feature? How do you prevent this during experiment design?
A concrete example
Suppose a targeting bug places 80 percent new users and 20 percent returning users into treatment, while control gets the inverse. New users baseline at 10 percent engagement and returning users at 50 percent. With the feature, new users rise to 12 percent and returning users rise to 55 percent, so the feature wins in every segment. Yet the treatment blend is roughly 20.6 percent while the control blend is 42 percent, producing an aggregate drop even though every subgroup improved.
Interview question
You observe that a feature improves engagement within every user segment but degrades it overall. Which explanation identifies the true statistical mechanism?
- a.The feature helps some segments and hurts others, causing effects to cancel in aggregate
- b.Random variance in the overall metric masks the consistent segment-level gains
- c.The aggregate is a weighted average skewed by unequal population mixes between groupsCorrect
- d.A confounder interacts with user type to create opposite effects in different segments
Why? this is the answer
Simpson's Paradox occurs when unequal segment sizes or traffic allocations skew the aggregate into a weighted average that reverses the direction seen in every subgroup. Option A describes an interaction effect, not Simpson's Paradox, because the subgroup directions differ rather than agreeing.
Just read this? Test yourself on what you have been reading.
Read the original → statsig.com
- #simpsons paradox
- #ab testing
- #metrics
- #experimentation
- #statistics
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles