When user-level A/B tests get contaminated
WHAT IT TESTS: recognizing interference that breaks the independence assumption. OUTLINE: network or marketplace spillover violates SUTVA, so randomize by cluster (geo, group, time) and analyze at that level.
WHAT IT TESTS: whether you understand when user-level randomization violates the independence assumption (SUTVA). ANSWER OUTLINE: describe a scenario where treated and control users interact, such as a social network, two-sided marketplace, or shared-resource feature, so one user's treatment leaks onto another's outcome. Explain that this contaminates the estimated effect. Propose a higher unit of randomization, such as cluster, geo, or time-based assignment, so interacting users land in the same arm.
Read the original → interview
- #ab-testing
- #interference
- #cluster-randomization
- #sutva
- #experiment-design
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.