Why is user-level randomization flawed by spillover effects?

WHAT THIS TESTS: Your understanding of second-order effects in experimentation, specifically network interference or spillover. Interviewers want to see if you can identify when a standard A/B test setup is invalid due to violations of the Stable Unit Treatment Value Assumption (SUTVA) and if you know the more advanced techniques required to get a valid result.

A GOOD ANSWER COVERS: First, define the problem. With user-level randomization, a user in the treatment group (e.g., gets new recommendations) might share content with a friend in the control group. This contaminates the control group, as their behavior is no longer independent of the treatment. This violates SUTVA and typically leads to an underestimation of the true treatment effect. Second, propose graph cluster randomization as the solution. This involves partitioning the entire user graph into many smaller, densely connected clusters. Third, explain the mechanism. Instead of randomizing users, you randomize entire clusters. All users within a single cluster are assigned to the same group (treatment or control), which minimizes the connections between the two populations and reduces spillover. Finally, discuss analysis and trade-offs. Acknowledge the complexity; you can't just compare user-level means. You need a weighted estimator, like a Horvitz-Thompson estimator, to calculate an unbiased average treatment effect. The trade-off is increased complexity for reduced bias.

COMMON WRONG ANSWERS: A major red flag is ignoring the core problem of interference. Some candidates suggest simply running the test for a shorter time or on a smaller population, which does not solve the contamination issue. Another weak answer is proposing ego-cluster randomization (randomizing a user and only their immediate friends) without discussing its limitations, as it fails to prevent spillover from friends-of-friends. The weakest answers fail to propose any graph-based solution at all.

LIKELY FOLLOW-UPS: "How would you actually create these clusters? What algorithms would you use?" (Mention community detection algorithms like Louvain or Girvan-Newman). "What are the trade-offs of this method versus a simple A/B test?" (Reduced bias vs. increased implementation complexity and potentially higher variance if clusters are poor). "How does the variance of your estimate change with this design?" (With proper clustering, variance can be exponentially lower than with naive randomization under interference).

ONE CONCRETE EXAMPLE: Imagine testing a new "share to feed" ranking algorithm. With user-level randomization, Alice (treatment) sees a video via the new algorithm and shares it. Her friend Bob (control) sees the shared video. Bob's engagement is now influenced by the treatment, but he's in the control group. The system incorrectly attributes his engagement to the baseline experience, diluting the measured effect. With graph cluster randomization, Alice and her entire community would be in the treatment group together, while Bob and his separate community would be in control, preventing this direct contamination.