Run concurrent experiments without interference
designing for parallel experiments via layered infrastructure.
independent non-interacting tests can share traffic through orthogonal layers; interacting ones need mutual exclusion in a shared layer.
What's really being asked
This probes whether you understand layered experimentation infrastructure, the standard way large companies run many tests at once. The interviewer wants you to reason about interaction risk and choose between orthogonal overlap and mutual exclusion rather than serializing everything.
The full answer
The first question is whether the onboarding test and the navigation test can interact. Onboarding affects new-user setup; navigation affects ongoing browsing. If their surfaces and measured metrics are effectively independent, you can run them orthogonally: each user is independently and deterministically randomized into a variant of each experiment using separate hash seeds per experiment. Across many users, the navigation variants are evenly distributed within each onboarding variant, so the two effects do not confound and you reclaim full traffic for both tests simultaneously. If, instead, two experiments could plausibly interact, for example both altering the same screen, you place them in the same layer so assignment is mutually exclusive and no user is ever in both. A layered framework formalizes this: a layer holds mutually exclusive experiments, and orthogonal layers overlap freely. Even with orthogonal designs, monitor for interaction effects by checking whether one experiment's effect differs across the other's variants.
The mistakes people make
Declaring all concurrent experiments must be mutually exclusive, which serializes the roadmap and wastes traffic. Running clearly interacting tests orthogonally and then trusting confounded results. Using the same hash seed for both experiments, which correlates assignments. Forgetting to monitor interaction effects entirely.
What usually comes next
How do you detect an unexpected interaction? Segment each experiment's results by the other's variant. When is mutual exclusion worth the traffic cost? When interaction is likely or stakes are high. How do layers map to feature flags and assignment services?
A concrete example
The onboarding test runs in layer one and the navigation test in orthogonal layer two. A user is hashed with the onboarding seed into the new flow and independently hashed with the navigation seed into the control menu. Thousands of users spread evenly across the four combinations, so each experiment is analyzed on full traffic. The team also slices the navigation result by onboarding variant and sees no interaction, confirming the orthogonal design held.
Interview question
Two experiments touch independent surfaces. Why can running them in orthogonal layers be valid while serving both on full traffic?
- a.It halves the required sample size for each experiment
- b.It removes the need to monitor for interaction effects
- c.Independent randomization spreads each experiment's variants evenly within the other, so effects do not confoundCorrect
- d.Orthogonal layers guarantee no two experiments can ever interact
Why? this is the answer
Independent per-experiment hashing distributes one test's variants evenly across the other's, so the effects stay separable and both use full traffic. Orthogonality assumes no interaction rather than guaranteeing it, so monitoring is still required.
Just read this? Test yourself on what you have been reading.
Read the original → research.google
- #experimentation
- #layered-infrastructure
- #orthogonal-experiments
- #mutual-exclusion
- #experiment-design
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on experimentation — each one lists the topics its interview covers.
See open roles