Experiment design under network effects

WHAT THIS TESTS This probes understanding of interference: classic experiments assume one unit's treatment does not affect another's outcome. Referral and social features break that, so the candidate must redesign randomization and analysis.

A GOOD ANSWER COVERS Name the violated assumption, often called SUTVA or no interference. With user-level randomization, a treated user who can invite friends may invite control users, so control outcomes are contaminated and the estimated effect shrinks toward zero. The fix is to randomize at the level of clusters that contain most interactions: connected components of the social graph, detected communities, geographies, or organizations. Assign whole clusters to treatment or control so spillover stays inside an arm. Analysis then treats the cluster as the unit, which reduces effective sample size and inflates variance, so you need enough clusters and methods like cluster-robust standard errors. Consider ego-network or switchback designs as alternatives.

COMMON WRONG ANSWERS Keeping user-level randomization and trusting the dilution. Ignoring that variance now lives at the cluster level and computing significance on individual users, overstating power. Forgetting that clustering reduces statistical power and undersizing the test.

LIKELY FOLLOW-UPS How do you build clusters when the graph is dense and components are huge? What is a switchback design and when is it better? How do you estimate the bias direction if you had run it naively?

ONE CONCRETE EXAMPLE A referral program is tested by partitioning users into 2000 social communities via graph clustering. Each community is randomly assigned wholly to treatment or control. Invites flow within a community, so spillover is contained. You compare mean signups per community across arms using cluster-level standard errors, accepting lower power for an unbiased estimate.