A/B test shows 5% lift with imbalanced mobile traffic. What next?
This tests sample ratio mismatch and Simpson's paradox. A strong answer says the lift is confounded by platform, demands stratified analysis to isolate the true effect, and checks randomization. A red flag is accepting the 5% lift without questioning skew.
This tests whether you recognize sample ratio mismatch and Simpson's paradox in online experiments. A strong answer explains that the 5% lift is likely confounded because mobile users often behave differently than desktop users, so the treatment effect is entangled with platform composition. It then proposes stratified analysis or reweighting to estimate the true effect per platform, checks for bugs in randomization or targeting, and recommends against shipping until the effect is isolated.
Read the original → statsig.com
- #ab-testing
- #experimentation
- #simpsons-paradox
- #sample-ratio-mismatch
- #causal-inference
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.