Why is user-level randomization flawed by spillover effects?
Tests your grasp of SUTVA violations in network experiments. Explain how spillover contaminates the control group, then propose graph cluster randomization—grouping users and assigning entire clusters to A/B variants—to minimize interference.
What's really being asked
Your understanding of second-order effects in experimentation, specifically network interference or spillover. Interviewers want to see if you can identify when a standard A/B test setup is invalid due to violations of the Stable Unit Treatment Value Assumption (SUTVA) and if you know the more advanced techniques required to get a valid result.
The full answer
First, define the problem. With user-level randomization, a user in the treatment group (e.g., gets new recommendations) might share content with a friend in the control group. This contaminates the control group, as their behavior is no longer independent of the treatment. This violates SUTVA and typically leads to an underestimation of the true treatment effect. Second, propose graph cluster randomization as the solution. This involves partitioning the entire user graph into many smaller, densely connected clusters. Third, explain the mechanism. Instead of randomizing users, you randomize entire clusters. All users within a single cluster are assigned to the same group (treatment or control), which minimizes the connections between the two populations and reduces spillover. Finally, discuss analysis and trade-offs. Acknowledge the complexity; you can't just compare user-level means. You need a weighted estimator, like a Horvitz-Thompson estimator, to calculate an unbiased average treatment effect. The trade-off is increased complexity for reduced bias.
The mistakes people make
A major red flag is ignoring the core problem of interference. Some candidates suggest simply running the test for a shorter time or on a smaller population, which does not solve the contamination issue. Another weak answer is proposing ego-cluster randomization (randomizing a user and only their immediate friends) without discussing its limitations, as it fails to prevent spillover from friends-of-friends. The weakest answers fail to propose any graph-based solution at all.
What usually comes next
"How would you actually create these clusters? What algorithms would you use?" (Mention community detection algorithms like Louvain or Girvan-Newman). "What are the trade-offs of this method versus a simple A/B test?" (Reduced bias vs. increased implementation complexity and potentially higher variance if clusters are poor). "How does the variance of your estimate change with this design?" (With proper clustering, variance can be exponentially lower than with naive randomization under interference).
A concrete example
Imagine testing a new "share to feed" ranking algorithm. With user-level randomization, Alice (treatment) sees a video via the new algorithm and shares it. Her friend Bob (control) sees the shared video. Bob's engagement is now influenced by the treatment, but he's in the control group. The system incorrectly attributes his engagement to the baseline experience, diluting the measured effect. With graph cluster randomization, Alice and her entire community would be in the treatment group together, while Bob and his separate community would be in control, preventing this direct contamination.
Interview question
When designing an A/B test for a new social feature, why is graph cluster randomization often preferred over standard user-level randomization?
- a.To isolate the treatment's effect by minimizing interaction between treatment and control users.Correct
- b.To decrease the experiment's complexity and the variance of the results.
- c.To ensure the treatment and control groups are of perfectly equal size.
- d.To allow for a simpler analysis using a direct comparison of user-level means.
Why? this is the answer
Graph cluster randomization is used to reduce bias from spillover effects, where the treatment group influences the control group. By assigning entire communities to one variant, it minimizes these interactions. A common distractor is assuming it simplifies analysis, but it actually requires more complex weighted estimators.
Just read this? Test yourself on what you have been reading.
Read the original → arxiv.org
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on experimentation — each one lists the topics its interview covers.
See open roles