Sample Ratio Mismatch (SRM): When Your A/B Test Is Broken

THE MENTAL MODEL: Sample Ratio Mismatch (SRM) is a critical alert that your A/B test is fundamentally broken. It means the number of users in your control and variation groups doesn't match the intended traffic split. This isn't just random noise; it signals a systematic bias in how users are assigned, invalidating the core assumption that the groups are comparable.

HOW IT WORKS: SRM is detected statistically using a Chi-Square Goodness of Fit test. This test compares the observed visitor counts in each variation to the expected counts. For example, in a 50/50 split with 100,000 total users, you'd expect 50,000 in each group. If you get 51,000 and 49,000, the test calculates the probability (p-value) that this deviation is due to random chance. If the p-value is below a strict threshold, like 0.001, it confirms an SRM.

WHEN TO USE IT: You should check for SRM in every A/B test as a mandatory health check before interpreting results. It's the first step in validating your experiment's integrity. On high-traffic sites, this is especially crucial, as even tiny percentage imbalances can represent thousands of users and lead to incorrect business decisions. Most modern A/B testing platforms automate this check.

WHEN NOT TO USE IT: An SRM alert doesn't mean you must immediately discard all your data. The first step is to pause the experiment to stop collecting biased traffic. Then, investigate the root cause. Use breakdown reports in your analytics tool to segment users by device type, browser, or operating system. Often, the mismatch is isolated to a specific segment. By filtering out this problematic segment, you can sometimes still analyze the remaining data for valid conclusions.

ONE CANONICAL EXAMPLE: A frequent cause of SRM is a client-side bug. Imagine your new variation uses a JavaScript feature that isn't supported by older browsers. When a user with an old browser is assigned to the variation, the script fails, and they are redirected to the original page. They were intended for the variation but are now counted in the control group. This inflates the control group's numbers and skews its composition to include more users with older browsers, breaking random assignment.