How do network effects violate A/B tests and how to mitigate them?

What's really being asked

This question evaluates whether you understand that standard A/B tests rely on the Stable Unit Treatment Value Assumption, which requires that one user's treatment does not affect another user's outcome. In social networks this assumption collapses because of interference, also called network effects, where a treated user changes the experience or behavior of connected control users. The interviewer wants to see that you can identify this violation, explain why it biases treatment effect estimates, and design a real-world experiment that accounts for graph-based spillover.

The full answer

First, define SUTVA and explain the specific violation. Colloquially, interference means a unit's response depends on other units' treatments, not just its own. On a social media platform, if a new messaging feature causes treated users to send more messages, their friends in the control group may receive and respond to more messages, contaminating the control bucket and compressing the measured effect. Second, propose cluster randomization instead of individual randomization. You should cluster users by the social graph so that most interactions fall inside clusters rather than across cluster boundaries. Third, note that the clustering itself is a design choice. Mention that imbalanced clusters often achieve a better bias-variance trade-off than balanced clusters because real social networks have highly skewed degree distributions. Fourth, discuss analysis improvements such as cluster-based regression adjustment and trigger logging to reduce variance and improve precision when estimating treatment effects.

The mistakes people make

A major red flag is insisting that user-level randomization is fine if the sample size is large enough, since volume does not fix structural bias from spillover. Another weak answer is proposing geographic clustering without explaining why geography maps to the interference mechanism; unless the feature is explicitly local, geographic borders rarely capture social graph edges. A third red flag is suggesting a switchback or time-based design without discussing the specific network structure, which may miss peer-to-peer dynamics entirely.

What usually comes next

The interviewer may ask how you would construct the clusters in practice, how you would validate that a given clustering actually captures interference, or how you would balance the need for many small clusters against the need to keep edges internal. They might also ask about power implications, since cluster randomization typically requires larger sample sizes, or about how to handle overlapping universes where multiple experiments run simultaneously on the same graph.

A concrete example

Consider a Stories experiment on a social media platform. If the treatment encourages users to post more Stories, their friends see more Story content in their feeds. With individual randomization, control users who are friends with treated users receive a diluted control experience, so the platform underestimates the true treatment effect. By clustering users based on dense social graph neighborhoods and randomizing entire clusters into treatment or control, the interference is largely contained within clusters. Meta has deployed this framework at scale and found that using imbalanced clusters plus cluster-based regression adjustment substantially improves both bias and precision compared to naive balanced clustering.