Why is stopping an A/B test when it hits significance problematic?
Tests your understanding of the 'peeking problem' in A/B testing. A great answer defines peeking, explains how it inflates the Type I error rate (false positives), and states the need for a predetermined sample size.
This question tests your practical grasp of A/B testing statistics, specifically the 'peeking problem.' A strong answer names the issue, explains that repeated checks increase the cumulative probability of a false positive (Type I error), and advocates for running tests to a pre-calculated sample size. A common red flag is giving a vague answer about 'unstable results' without explaining the underlying statistical inflation of the error rate, or confusing peeking with the multiple comparisons problem.
Read the original → docs.growthbook.io
- #a/b testing
- #statistics
- #metrics
- #product analytics
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.