Why is user-level randomization flawed by spillover and what is an alternative?

What's really being asked

This question tests whether you understand that the Stable Unit Treatment Value Assumption (SUTVA) is often violated in social networks. In standard A/B tests, we assume one user's treatment does not affect another's outcome. On a social platform, a new recommendation algorithm shown to a treated user changes what they share, like, or comment, which then spills over to their neighbors in the control group. The interviewer wants to see if you can identify this interference, explain why it biases the average treatment effect, and propose a principled design that accounts for the underlying graph structure.

The full answer

A strong answer hits four things in order. First, define the spillover problem: when treatment effects propagate along edges, the control group is contaminated, so the naive difference in means no longer estimates the true causal effect. Second, name the alternative: graph cluster randomization. Third, explain the mechanics: partition the user graph into clusters, randomize entire clusters into treatment or control, and define network exposure conditions based on a user's cluster and neighborhood. Fourth, discuss estimation: use Horvitz-Thompson inverse probability weighting where weights are the inverse of each vertex's probability of being network exposed under the cluster randomization scheme. Optionally, note that under a restricted growth condition on neighborhood sizes, this design yields variance that scales linearly with degree rather than exponentially, making it feasible for large graphs.

The mistakes people make

The biggest red flag is suggesting you can fix this by dropping edges, removing highly connected users, or running a standard user-level A/B test and ignoring the graph. Another weak answer is proposing to randomize at the session level without defining exposure or accounting for the fact that neighbors still interact. Some candidates mention difference-in-differences or synthetic control; these do not solve the fundamental SUTVA violation here. Finally, do not confuse cluster randomization with simple geographic or demographic stratification; the clusters must be derived from the network topology itself.

What usually comes next

An interviewer might ask how you would compute network exposure probabilities in practice, or how to choose the clustering algorithm. They may probe the bias-variance tradeoff: larger clusters reduce interference but increase variance because fewer units are randomized. They could ask what happens if the exposure model is misspecified, or how you would validate that spillovers actually exist using observational data before running the experiment.

A concrete example

Imagine a platform with 10 million users where the median user has 200 friends. In a user-level randomization, a treated user who shares 30 percent more viral content because of the new algorithm exposes roughly half of those friends to the content, meaning many control users are indirectly treated. If you instead partition the graph into clusters of roughly 1,000 users using a neighborhood-based clustering algorithm, assign whole clusters to treatment or control, and weight each user by their inverse probability of network exposure, you can recover an unbiased average treatment effect. Under restricted growth, the estimator variance grows linearly with the 200 median degree rather than exponentially, making the experiment statistically viable.