Describe cluster or switchback randomization for network-effect A/B tests.

WHAT THIS TESTS: This question evaluates whether you understand that the Stable Unit Treatment Value Assumption is violated when users interact, making standard A/B tests biased. It tests your ability to design valid alternatives and anticipate the real engineering costs of moving away from user-level assignment.

A GOOD ANSWER COVERS: A strong response starts by naming two viable strategies. First is cluster randomization, where you assign treatment at the level of geography, workplace, organization, or social cluster so that spillover stays within the treated group. Second is switchback testing, where you alternate treatment and control across time windows for the same population, using time as the unit of randomization. Then it details engineering challenges. The routing layer must change from user-scoped to cluster-scoped or time-scoped bucketing, which breaks existing SDK assumptions. Power drops sharply because you have far fewer independent units, often inflating required sample size by one to two orders of magnitude. You must handle correlated standard errors because observations within a cluster or time window are not independent, requiring clustered standard errors or mixed-effects models. Operationally, switchback demands idempotent feature toggling and cache warming to handle rapid flips, while cluster randomization requires stable cluster definitions and handling edge cases where users move between clusters.

COMMON WRONG ANSWERS: The biggest red flag is suggesting user-level randomization with post-hoc regression adjustments for network exposure, which does not recover valid causal estimates under interference. Another failure mode is proposing graph-cluster randomization without acknowledging the engineering cost of computing and maintaining dynamic social clusters at query time. Candidates also stumble by ignoring the power implications, treating a shift from millions of users to hundreds of clusters as a minor statistical detail.

LIKELY FOLLOW-UPS: An interviewer might ask how you would choose between cluster and switchback given your product structure, or how you would detect network effects before committing to a complex design. They may probe how you handle carryover effects in switchback experiments, or ask how you analyze results when you have only dozens of clusters but millions of observations. Be ready to discuss variance estimation under few clusters and whether bootstrap or permutation methods are appropriate.

ONE CONCRETE EXAMPLE: Consider a workplace collaboration tool where teams chat in shared channels. If you randomize at the user level, a treated user posting more messages floods control teammates with notifications, contaminating the control experience. Instead, you cluster randomize at the team level. The engineering team must refactor the experiment assignment service to accept a team ID rather than a user ID, update the data pipeline to emit team-level exposure logs, and accept that with five thousand teams you need a six-month run instead of a one-week run to reach equivalent power. Alternatively, a ride-hailing marketplace could use switchback testing, turning a new pricing algorithm on for two-hour windows and off for the next two. Engineers must ensure the pricing service can switch rapidly without cold-start latency, and analysts must model day-of-week seasonality to avoid confounding.

Source: statsig.com