Peeking in A/B tests and how to mitigate it
WHAT IT TESTS: understanding inflated false positives from repeated looks. OUTLINE: peeking is checking significance repeatedly and stopping at the first significant result, which inflates false positives; mitigate with fixed sample sizes or sequential…
WHAT IT TESTS: whether you understand that repeatedly testing significance and stopping when it first crosses the threshold dramatically inflates the false-positive rate. ANSWER OUTLINE: define peeking, explain why each look is another chance to cross 0.05 by luck so the true error rate far exceeds the nominal one, and give mitigations: pre-compute a fixed sample size and decide only at the end, or use methods designed for continuous monitoring like sequential testing or always-valid confidence intervals.
Read the original → interview
- #ab-testing
- #peeking
- #statistical-significance
- #sequential-testing
- #false-positives
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.