What is the multiple comparisons problem in UX research?

WHAT THIS TESTS: The interviewer wants to see that you understand how statistical validity erodes when many hypotheses are tested simultaneously. In UX research, teams routinely run A/B tests with multiple metrics, compare several design variants, or iterate on tests until something looks significant. The question checks whether you recognize that each test carries its own Type I error rate and that the family-wise error rate grows with the number of tests. It also probes whether you can move beyond textbook definitions and apply a correction in a practical research context.

A GOOD ANSWER COVERS: First, define the multiple comparisons problem as the increased probability of making at least one false positive when many statistical inferences are performed on the same dataset. Second, describe a realistic UX scenario such as an A/B test that tracks twenty success metrics or a checkout flow experiment that compares five different button colors against a control; in both cases, the chance of a spurious win rises simply because you looked more than once. Third, name at least one correction method and explain when to use it. The classic Bonferroni adjustment divides the alpha level by the number of tests, which controls the family-wise error rate but is conservative when tests are numerous. The Benjamini-Hochberg procedure controls the false discovery rate and offers more power when you are screening many metrics and can tolerate a small proportion of false positives. Fourth, mention practical non-statistical guards such as pre-registration of primary metrics or splitting data into exploratory and confirmatory sets.

COMMON WRONG ANSWERS: A major red flag is insisting that every p-value below 0.05 is a real finding regardless of how many tests were run. Another weak response is mentioning Bonferroni without understanding its cost; saying you will always use it can signal rote memorization rather than judgment. Confusing Type I and Type II error is also damaging. Finally, suggesting that running tests sequentially rather than simultaneously avoids the problem is incorrect; peeking and optional stopping inflate error rates in a similar way.

LIKELY FOLLOW-UPS: The interviewer may ask how you would prioritize one primary metric versus twenty secondary metrics. They might probe the difference between family-wise error rate control and false discovery rate control. You could also be asked how Bayesian methods or sequential testing platforms affect the need for classical multiplicity corrections. Be ready to discuss whether correction is even necessary for purely exploratory research.

ONE CONCRETE EXAMPLE: Imagine a redesign of a mobile onboarding flow. The team runs an A/B test measuring completion rate, time to complete, drop-off at each of five steps, and satisfaction score. That is eight tests. At an uncorrected alpha of 0.05, the probability of at least one false positive is roughly 1 minus 0.95 to the eighth power, or about 34 percent. To protect decision quality, you pre-register completion rate as the primary metric and apply no correction to it alone. For the seven secondary metrics, you use the Benjamini-Hochberg procedure at a false discovery rate of 0.10. This balances discovery against reliability and shows the interviewer that you can design a credible testing protocol.

Source: Wikipedia: Multiple comparisons problem