Handling spillover effects in social network A/B tests

What's really being asked

This question probes your understanding of the limitations of standard A/B testing, specifically the failure of the Stable Unit Treatment Value Assumption (SUTVA) in connected systems. The interviewer is testing if you can diagnose why user-level randomization is statistically invalid in a social network context and if you can propose a more sophisticated and appropriate experimental design. It distinguishes candidates who only know basic randomization from senior engineers who grasp the nuances of causal inference in complex environments.

The full answer

First, define the problem: user-level randomization violates SUTVA because a user's outcome is influenced by the treatment assignment of their peers. Second, explain the mechanism of this 'spillover' or 'interference': a user in the treatment group shares content or takes an action visible to their friends in the control group, thus 'contaminating' the control. Third, state the impact: this contamination artificially inflates the metric in the control group, leading to a systematic underestimation of the true treatment effect. You might discard a valuable feature. Fourth, propose a solution: graph cluster randomization. You partition the social graph into communities (clusters) and then randomize entire clusters into treatment or control. This minimizes the network edges between A and B, containing the spillover. For a top answer, mention this requires advanced analysis like a Horvitz-Thompson estimator to produce an unbiased effect estimate.

The mistakes people make

A major red flag is suggesting randomization techniques that don't account for the network structure. For example, randomizing by user ID hash, geographic location, or sign-up date. While these are valid segmentation strategies for other problems, they do nothing to prevent friends from being in different experimental groups. Another weak answer is being vague about the impact, saying results will be 'messy' or 'inaccurate' without specifying that the effect will be consistently underestimated. Finally, suggesting a switchback or time-based randomization is incorrect for this problem; it addresses different issues (like marketplace liquidity) and is vulnerable to temporal confounds while not solving user-to-user interference.

What usually comes next

Expect questions like, 'How would you technically implement graph clustering? What algorithms?' (e.g., Louvain, Girvan-Newman). Another common follow-up is, 'What are the downsides of this approach?' (e.g., increased variance, requiring larger sample sizes or effect sizes for significance; much higher implementation complexity). You might also be asked about choosing the cluster size, which involves a trade-off between reducing bias (larger clusters are better) and minimizing variance (more, smaller clusters are better).

A concrete example

Imagine testing a new recommendation algorithm on a social feed. With user-level randomization, User A (treatment) sees and shares a post from the new algorithm. Their friend, User B (control), now sees this shared post in their own feed. User B's engagement is now influenced by the treatment, contaminating the control group. The measured lift in engagement for the new algorithm will be diluted because the control group's engagement is artificially higher than a true baseline. Using graph cluster randomization, we would identify a community of strongly connected users and place them all in the same group (A or B), preventing User A's activity from directly spilling over to User B.