Sample Ratio Mismatch (SRM): When Your A/B Test Is Broken
Sample Ratio Mismatch (SRM) means your A/B test's traffic split is broken, violating random assignment. For example, a 50/50 split results in a statistically significant imbalance.
The mental model
Sample Ratio Mismatch (SRM) is a critical alert that your A/B test is fundamentally broken. It means the number of users in your control and variation groups doesn't match the intended traffic split. This isn't just random noise; it signals a systematic bias in how users are assigned, invalidating the core assumption that the groups are comparable.
How it works
SRM is detected statistically using a Chi-Square Goodness of Fit test. This test compares the observed visitor counts in each variation to the expected counts. For example, in a 50/50 split with 100,000 total users, you'd expect 50,000 in each group. If you get 51,000 and 49,000, the test calculates the probability (p-value) that this deviation is due to random chance. If the p-value is below a strict threshold, like 0.001, it confirms an SRM.
When to use it
You should check for SRM in every A/B test as a mandatory health check before interpreting results. It's the first step in validating your experiment's integrity. On high-traffic sites, this is especially crucial, as even tiny percentage imbalances can represent thousands of users and lead to incorrect business decisions. Most modern A/B testing platforms automate this check.
When not to use it
An SRM alert doesn't mean you must immediately discard all your data. The first step is to pause the experiment to stop collecting biased traffic. Then, investigate the root cause. Use breakdown reports in your analytics tool to segment users by device type, browser, or operating system. Often, the mismatch is isolated to a specific segment. By filtering out this problematic segment, you can sometimes still analyze the remaining data for valid conclusions.
One canonical example
A frequent cause of SRM is a client-side bug. Imagine your new variation uses a JavaScript feature that isn't supported by older browsers. When a user with an old browser is assigned to the variation, the script fails, and they are redirected to the original page. They were intended for the variation but are now counted in the control group. This inflates the control group's numbers and skews its composition to include more users with older browsers, breaking random assignment.
Interview question
Which of the following is the most critical implication of a detected Sample Ratio Mismatch (SRM) in an A/B test?
- a.The experiment's primary metric will inevitably show a false positive result.
- b.The fundamental assumption of random assignment between groups has been violated.Correct
- c.The statistical power of the test is insufficient to detect a meaningful effect.
- d.The A/B testing platform used is inherently flawed and unreliable.
Why? this is the answer
SRM is a critical alert that the random assignment of users to experiment groups has failed, invalidating the core assumption that the groups are comparable. While SRM does invalidate results, it doesn't guarantee a false positive; it simply means any observed differences are unreliable and untrustworthy.
Just read this? Test yourself on what you have been reading.
Read the original → help.kameleoon.com
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on a/b testing — each one lists the topics its interview covers.
See open roles