Why repeatedly extending a test inflates false positives
WHAT IT TESTS: understanding the peeking problem and inflated false positives. OUTLINE: repeatedly checking and extending until significance is p-hacking via optional stopping, which inflates the false-positive rate; fix with fixed sample sizes or sequential…
WHAT IT TESTS: whether you recognize optional stopping as a statistical hazard. ANSWER OUTLINE: name it the peeking problem, a form of p-hacking. Fixed-horizon significance tests assume a sample size chosen in advance; repeatedly checking and extending until you hit significance multiplies the chances of a spurious result, inflating the false-positive rate well above the nominal level.
Read the original → interview
- #peeking
- #p-hacking
- #optional-stopping
- #statistical-significance
- #sequential-testing
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.