Interpret a p-value in an A/B test
WHAT IT TESTS: correct meaning of a p-value. OUTLINE: define it as P(data this extreme | null true), interpret 0.03 against a 0.05 threshold, and state what it is NOT. RED FLAG: claiming a 3% chance the result is wrong or that the effect is.
WHAT IT TESTS: whether you truly understand statistical significance rather than reciting it. ANSWER OUTLINE: a p-value is the probability of observing a result at least as extreme as yours assuming the null hypothesis (no difference) is true. A p-value of 0.03 means that if the variants were actually identical, you would see a gap this large only about 3 percent of the time, so at a 0.05 threshold you reject the null. RED FLAG: saying there is a 3 percent chance the result is due to chance or that the variant is wrong, or.
Read the original → interview
- #p-value
- #ab-testing
- #statistical-significance
- #hypothesis-testing
- #experimentation
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.