Run concurrent experiments without interference

WHAT THIS TESTS This probes whether you understand layered experimentation infrastructure, the standard way large companies run many tests at once. The interviewer wants you to reason about interaction risk and choose between orthogonal overlap and mutual exclusion rather than serializing everything.

A GOOD ANSWER COVERS The first question is whether the onboarding test and the navigation test can interact. Onboarding affects new-user setup; navigation affects ongoing browsing. If their surfaces and measured metrics are effectively independent, you can run them orthogonally: each user is independently and deterministically randomized into a variant of each experiment using separate hash seeds per experiment. Across many users, the navigation variants are evenly distributed within each onboarding variant, so the two effects do not confound and you reclaim full traffic for both tests simultaneously. If, instead, two experiments could plausibly interact, for example both altering the same screen, you place them in the same layer so assignment is mutually exclusive and no user is ever in both. A layered framework formalizes this: a layer holds mutually exclusive experiments, and orthogonal layers overlap freely. Even with orthogonal designs, monitor for interaction effects by checking whether one experiment's effect differs across the other's variants.

COMMON WRONG ANSWERS Declaring all concurrent experiments must be mutually exclusive, which serializes the roadmap and wastes traffic. Running clearly interacting tests orthogonally and then trusting confounded results. Using the same hash seed for both experiments, which correlates assignments. Forgetting to monitor interaction effects entirely.

LIKELY FOLLOW-UPS How do you detect an unexpected interaction? Segment each experiment's results by the other's variant. When is mutual exclusion worth the traffic cost? When interaction is likely or stakes are high. How do layers map to feature flags and assignment services?

ONE CONCRETE EXAMPLE The onboarding test runs in layer one and the navigation test in orthogonal layer two. A user is hashed with the onboarding seed into the new flow and independently hashed with the navigation seed into the control menu. Thousands of users spread evenly across the four combinations, so each experiment is analyzed on full traffic. The team also slices the navigation result by onboarding variant and sees no interaction, confirming the orthogonal design held.