A/B test shows 5% lift with imbalanced mobile traffic. What next?

WHAT THIS TESTS: This question tests three things: sensitivity to sample ratio mismatch, understanding of confounding variables in randomized experiments, and practical judgment about when to trust an aggregate metric. Senior candidates are expected to immediately flag that a 30-point platform skew breaks the randomization assumption and can create Simpson's paradox, where the aggregate trend reverses within subgroups.

A GOOD ANSWER COVERS: A good answer hits four things in order. First, invalidate the headline number: state clearly that the 5% lift is uninterpretable because treatment and control are not comparable. Second, hypothesize the driver: note that mobile users typically have different conversion or engagement rates than desktop users, so the observed lift may reflect platform mix rather than the feature. Third, propose a fix: recommend stratifying the analysis by platform to compute the treatment effect within mobile and within desktop separately, or use reweighting methods like inverse probability weighting to simulate balanced groups. Fourth, investigate the root cause: check for bugs in the assignment pipeline, targeting rules, or device detection that caused the skew, and verify that other covariates like geography or user tier are also balanced.

COMMON WRONG ANSWERS: Common wrong answers include saying the result is still valid because the total sample size is large, suggesting you simply normalize the metric by platform without stratifying the variance, or proposing to run the test longer to balance the groups over time. Another red flag is claiming a t-test or p-value already accounts for the imbalance. These answers reveal a rote understanding of statistics without experimental design intuition.

LIKELY FOLLOW-UPS: Interviewers often push deeper with follow-ups. They may ask how you would weight the per-platform effects back into a single decision metric, or what you would do if the treatment wins on desktop but loses on mobile. They might also ask how you would design the next experiment to prevent this, such as using stratified randomization or blocking on platform.

ONE CONCRETE EXAMPLE: Imagine mobile users convert at 2% and desktop at 4%. If treatment accidentally enrolled 80% mobile while control enrolled 50% mobile, the control group would have a higher blended conversion rate purely from its desktop mix. A stratified analysis might reveal the feature actually hurts mobile conversion by 1% and helps desktop by 1%, yielding a net negative or flat effect that was masked by the skew. In practice, you would pause the readout, debug the assignment logic, and require a clean rerun or a corrected analysis before any launch decision.

Source: statsig.com