A/B test shows 5% lift with imbalanced mobile traffic. What next?
This tests sample ratio mismatch and Simpson's paradox. A strong answer says the lift is confounded by platform, demands stratified analysis to isolate the true effect, and checks randomization. A red flag is accepting the 5% lift without questioning skew.
What's really being asked
This question tests three things: sensitivity to sample ratio mismatch, understanding of confounding variables in randomized experiments, and practical judgment about when to trust an aggregate metric. Senior candidates are expected to immediately flag that a 30-point platform skew breaks the randomization assumption and can create Simpson's paradox, where the aggregate trend reverses within subgroups.
The full answer
A good answer hits four things in order. First, invalidate the headline number: state clearly that the 5% lift is uninterpretable because treatment and control are not comparable. Second, hypothesize the driver: note that mobile users typically have different conversion or engagement rates than desktop users, so the observed lift may reflect platform mix rather than the feature. Third, propose a fix: recommend stratifying the analysis by platform to compute the treatment effect within mobile and within desktop separately, or use reweighting methods like inverse probability weighting to simulate balanced groups. Fourth, investigate the root cause: check for bugs in the assignment pipeline, targeting rules, or device detection that caused the skew, and verify that other covariates like geography or user tier are also balanced.
The mistakes people make
Common wrong answers include saying the result is still valid because the total sample size is large, suggesting you simply normalize the metric by platform without stratifying the variance, or proposing to run the test longer to balance the groups over time. Another red flag is claiming a t-test or p-value already accounts for the imbalance. These answers reveal a rote understanding of statistics without experimental design intuition.
What usually comes next
Interviewers often push deeper with follow-ups. They may ask how you would weight the per-platform effects back into a single decision metric, or what you would do if the treatment wins on desktop but loses on mobile. They might also ask how you would design the next experiment to prevent this, such as using stratified randomization or blocking on platform.
A concrete example
Imagine mobile users convert at 2% and desktop at 4%. If treatment accidentally enrolled 80% mobile while control enrolled 50% mobile, the control group would have a higher blended conversion rate purely from its desktop mix. A stratified analysis might reveal the feature actually hurts mobile conversion by 1% and helps desktop by 1%, yielding a net negative or flat effect that was masked by the skew. In practice, you would pause the readout, debug the assignment logic, and require a clean rerun or a corrected analysis before any launch decision.
Interview question
When an A/B test shows a 5% aggregate lift but treatment has 80% mobile users versus 50% in control, what is the most appropriate next step?
- a.Weight the aggregate metric by platform share but skip separate per-platform treatment effect estimates
- b.Keep the test running longer until the platform mix naturally equalizes between treatment and control
- c.Stratify the analysis by platform and debug the assignment pipeline before trusting any lift estimateCorrect
- d.Trust the 5% lift because the total sample size is large enough to withstand the platform imbalance
Why? this is the answer
Stratifying by platform isolates the true treatment effect from the confounding platform skew, and debugging the assignment pipeline identifies the root cause. The most tempting distractor—trusting the lift because the sample size is large—is wrong because sample ratio mismatch breaks randomization no matter how large N is.
Just read this? Test yourself on what you have been reading.
Read the original → statsig.com
- #ab-testing
- #experimentation
- #simpsons-paradox
- #sample-ratio-mismatch
- #causal-inference
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on ab-testing — each one lists the topics its interview covers.
See open roles