Why is stopping an A/B test early problematic?
Tests understanding of the 'peeking problem' in A/B testing. A good answer defines peeking, explains how it inflates false positive rates, and contrasts it with waiting for a pre-determined sample size. A red flag is not explaining the statistical mechanism.
This tests your grasp of the 'peeking problem' in A/B testing and its impact on statistical validity, showing you can guide product decisions away from unsound practices. A great answer first names the issue ('peeking'), explains how each check inflates the false positive rate, and contrasts this with the correct method: calculating a sample size in advance and waiting until it's reached. A common mistake is simply saying 'it's bad' without explaining that the cumulative probability of a false positive increases with each check.
Read the original → docs.growthbook.io
- #a/b testing
- #statistics
- #analytics
- #metrics
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.