Prevent conflicting experiments with layers

What's really being asked

This checks whether you can enforce mutual exclusion structurally rather than by convention. The interviewer wants the layers or domains concept and a clear mechanism that makes it impossible, not merely discouraged, for a user to land in two experiments that fight over the same UI.

The full answer

The core idea is a layer: a partition of the traffic space dedicated to a group of experiments. All experiments that modify the same surface, here the checkout page, are placed in a single layer. Within that layer, each user is deterministically hashed, using the layer id and user id, into one of many buckets, and those buckets are allocated across the experiments in the layer. Because a user maps to exactly one bucket per layer, they are enrolled in at most one experiment from that layer, guaranteeing mutual exclusion for the checkout surface. Experiments on unrelated surfaces live in different, orthogonal layers, and a user is hashed independently per layer, so those experiments can overlap freely and reclaim full traffic. Domains generalize this: a domain can contain multiple layers, letting you carve out a slice of traffic for a set of mutually exclusive experiments while the rest of traffic runs orthogonally. The system, not a human, enforces the exclusion at assignment time.

The mistakes people make

Relying on a spreadsheet or team agreement to avoid overlaps, which fails as scale grows. Placing conflicting checkout experiments in orthogonal layers, so users get both and results are confounded. Using a single global mutual-exclusion rule for every experiment, which starves the roadmap of traffic. Hashing all experiments with the same seed, correlating assignments.

What usually comes next

How do you decide which experiments share a layer? Group by the surface or metric they could jointly affect. How do domains let you mix exclusive and orthogonal tests? Carve traffic into domains, each with its own layers. What happens to traffic allocation when a layer fills up?

A concrete example

Two proposed checkout experiments, a new button layout and a new payment-step order, are both assigned to the checkout layer. A user is hashed within that layer into bucket 37, which belongs to the button-layout experiment, so they can only ever see that one; the payment-step experiment draws from different buckets. Meanwhile a search-results experiment sits in an orthogonal layer, so the same user can simultaneously and independently participate in it.