An A/B test has imbalanced traffic. What do you do?

This tests your ability to spot confounding variables and Simpson's Paradox. A good answer first invalidates the aggregate result, then proposes segmenting by device to salvage insights, and finally investigates the root cause.
This tests your ability to identify confounding variables and Simpson's Paradox in A/B testing. A great answer immediately flags the result as invalid due to the imbalanced device mix. You should then outline a plan: first, segment the results by device type (mobile vs. desktop) to understand the true effect on each group. Second, launch an investigation into the root cause of the faulty randomization. Finally, decide whether to re-run the test based on the segmented analysis. A red flag is trying to justify the 5% lift without segmentation.
Read the original → statsig.com
- #a/b testing
- #analytics
- #statistics
- #product sense
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.