An A/B test has imbalanced traffic. What do you do?

WHAT THIS TESTS: This question tests your understanding of experimental validity beyond just looking at a p-value. The interviewer is checking if you can identify confounding variables, understand the concept of Simpson's Paradox (even if you don't use the name), and apply a rigorous, step-by-step diagnostic process. It separates candidates who just consume dashboards from those who can critically analyze and debug data integrity issues. Your ability to be pragmatic (salvaging data) versus dogmatic (immediately re-running) is also being evaluated.

A GOOD ANSWER COVERS: A strong answer follows a clear, four-step process. First, immediately invalidate the aggregate 5% lift. State clearly that the result is not trustworthy due to a sample ratio mismatch caused by a confounding variable (device type). Second, propose a diagnostic analysis. Before discarding the experiment, segment the results. Compare the metric lift for mobile users in treatment vs. control, and do the same for desktop users. This may reveal the true, un-skewed impact. Third, address the root cause. A traffic imbalance of this magnitude (80/20 vs 50/50) points to a severe bug in the randomization or assignment logic. This requires a high-priority technical investigation to protect all future experiments. Fourth, provide a recommendation. Based on the segmented analysis, advise on a course of action. This could be shipping the feature to a specific segment, iterating on the design, or re-running the test after the bucketing logic is fixed.

COMMON WRONG ANSWERS: The biggest red flag is accepting the 5% lift at face value, perhaps with a minor note about the imbalance. This shows a critical lack of statistical rigor. Another common mistake is immediately demanding a re-run of the experiment without first attempting to segment the data. This is inefficient and throws away potentially valuable, salvageable information. A third weak answer focuses only on the technical bug investigation while ignoring the immediate product decision that needs to be made about the feature itself. You must address both the data and the product.

LIKELY FOLLOW-UPS: Expect follow-ups that test your product sense and technical depth. For example: "What if your segmented analysis shows a +10% lift on mobile but a -15% lift on desktop? What do you recommend we do?" Or, "How would you technically investigate the source of this traffic imbalance? What are the first three things you would check?" Another good one is, "If we can't fix the root cause this quarter, is there any way to run a trustworthy experiment on this feature?" (Hint: stratified sampling or separate device-specific experiments).

ONE CONCRETE EXAMPLE: Imagine the metric is 'average items purchased'. The overall +5% lift is misleading. After segmentation, you might find mobile users went from 1.0 to 1.1 items (+10%), while desktop users went from 1.0 to 0.85 items (-15%). The aggregate +5% lift was an illusion created because the high-performing mobile segment was overrepresented in the treatment group (80% of users). The feature is actually harmful on desktop. The correct decision is not to launch, and to investigate why the desktop experience is so poor.