What is the multiple comparisons problem in UX research?
This tests whether you know running many tests inflates false positives. Strong answers define family-wise error, give a UX example like comparing twenty metrics in one A/B test, and name a correction like Bonferroni.
What's really being asked
The interviewer wants to see that you understand how statistical validity erodes when many hypotheses are tested simultaneously. In UX research, teams routinely run A/B tests with multiple metrics, compare several design variants, or iterate on tests until something looks significant. The question checks whether you recognize that each test carries its own Type I error rate and that the family-wise error rate grows with the number of tests. It also probes whether you can move beyond textbook definitions and apply a correction in a practical research context.
The full answer
First, define the multiple comparisons problem as the increased probability of making at least one false positive when many statistical inferences are performed on the same dataset. Second, describe a realistic UX scenario such as an A/B test that tracks twenty success metrics or a checkout flow experiment that compares five different button colors against a control; in both cases, the chance of a spurious win rises simply because you looked more than once. Third, name at least one correction method and explain when to use it. The classic Bonferroni adjustment divides the alpha level by the number of tests, which controls the family-wise error rate but is conservative when tests are numerous. The Benjamini-Hochberg procedure controls the false discovery rate and offers more power when you are screening many metrics and can tolerate a small proportion of false positives. Fourth, mention practical non-statistical guards such as pre-registration of primary metrics or splitting data into exploratory and confirmatory sets.
The mistakes people make
A major red flag is insisting that every p-value below 0.05 is a real finding regardless of how many tests were run. Another weak response is mentioning Bonferroni without understanding its cost; saying you will always use it can signal rote memorization rather than judgment. Confusing Type I and Type II error is also damaging. Finally, suggesting that running tests sequentially rather than simultaneously avoids the problem is incorrect; peeking and optional stopping inflate error rates in a similar way.
What usually comes next
The interviewer may ask how you would prioritize one primary metric versus twenty secondary metrics. They might probe the difference between family-wise error rate control and false discovery rate control. You could also be asked how Bayesian methods or sequential testing platforms affect the need for classical multiplicity corrections. Be ready to discuss whether correction is even necessary for purely exploratory research.
A concrete example
Imagine a redesign of a mobile onboarding flow. The team runs an A/B test measuring completion rate, time to complete, drop-off at each of five steps, and satisfaction score. That is eight tests. At an uncorrected alpha of 0.05, the probability of at least one false positive is roughly 1 minus 0.95 to the eighth power, or about 34 percent. To protect decision quality, you pre-register completion rate as the primary metric and apply no correction to it alone. For the seven secondary metrics, you use the Benjamini-Hochberg procedure at a false discovery rate of 0.10. This balances discovery against reliability and shows the interviewer that you can design a credible testing protocol.
Interview question
When an A/B test evaluates twenty success metrics independently at α = 0.05, what happens to the chance of at least one false positive?
- a.It is offset by a decreased Type II error rate since more metrics improve overall power
- b.It stays near 5% as long as the tests are run sequentially rather than simultaneously
- c.It remains at 5% because each test has its own independent alpha level
- d.It rises substantially above 5% as the family-wise error rate increasesCorrect
Why? this is the answer
Testing many metrics inflates the family-wise error rate, so the probability of at least one false positive grows well beyond the nominal 5%. Option C is tempting because each individual test does use α = 0.05, but that ignores the accumulation of error across the family of tests.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #ux research
- #statistics
- #ab testing
- #type i error
- #data literacy
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on ux research — each one lists the topics its interview covers.
See open roles