What is the 'multiple comparisons problem' in A/B testing?
Tests your grasp of statistical pitfalls in large-scale A/B testing. Define the problem (inflated false positives), explain the business risk (wasted effort), and propose a mitigation like Bonferroni correction.
This tests your grasp of statistical rigor in a high-velocity experimentation environment. A great answer defines the multiple comparisons problem (inflated false positives), quantifies the risk (20 tests at p=0.05 gives a >60% chance of a false positive), and proposes a mitigation like the Benjamini-Hochberg procedure or system-level controls. A common red flag is proposing overly conservative solutions like running only one test at a time, which ignores business needs for velocity and demonstrates a lack of practical experience.
Read the original → statsig.com
- #a/b testing
- #statistics
- #system design
- #metrics
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.