Design a multivariate experimentation platform with collision-free concurrent bucketing and cross-device consistency
Tests orthogonal layers and cross-session assignment persistence. Cover: deterministic hashing per layer, a user profile service for sticky bucketing, and stable ID resolution across devices. Red flag: random bucketing or local storage breaking consistency.
What's really being asked
This tests your ability to design a growth platform core primitive: assigning users to variations deterministically, in parallel, and persistently. Interviewers care about four things: statistical independence between experiments, cross-device consistency, identity resolution, and operational safety at scale. They want to see you understand that bucketing is not random sampling but a deterministic function of user identity and experiment configuration.
The full answer
First, orthogonal layers or namespaces. Each experiment lives in a layer identified by a layer ID. The bucketing hash is computed as hash(user_id + layer_id + salt) modulo 10000, which guarantees that Experiment A in Layer 1 and Experiment B in Layer 2 are independent and cannot steal traffic from each other. For multivariate tests, each factor combination is a variation and traffic is allocated across the full factorial grid.
Second, a user profile service. When a user is bucketed, the decision tuple (experiment_id, variation_id, timestamp) is written to a fast lookup store keyed by the resolved user ID. On subsequent requests, the service reads before re-computing the hash, ensuring the user never flips variations. This is how Optimizely implements sticky bucketing.
Third, identity resolution. The system evaluates a hierarchy: authenticated user ID is primary, then a stable bucketing ID from a first-party cookie or device ID, then a probabilistic ID. When a user logs in, the anonymous profile is merged into the authenticated profile, preserving existing assignments and avoiding double-counting.
Fourth, audience segmentation before bucketing. Dynamic segments are evaluated using attributes passed at decision time. If a user qualifies, they enter the experiment; if not, they receive the default and are not counted in the denominator. This prevents dilution of statistical power.
The mistakes people make
Using a single global hash space for all experiments causes traffic collisions and interaction effects. Relying on random number generators without a seed breaks reproducibility. Storing assignments only in browser localStorage fails cross-device and incognito scenarios. Re-bucketing on every evaluation destroys experiment validity and user trust. Ignoring the merge of anonymous to authenticated profiles inflates user counts and corrupts metrics.
What usually comes next
How do you handle mutual exclusion when two experiments in the same layer must not overlap? How do you implement a global holdout group that sees no experiments? What happens if the user profile service is down: do you fail open or closed? How do you prevent a bot from polluting the control or treatment? How do you support server-side rendering without a flash of wrong variation?
A concrete example
Imagine a SaaS product running three experiments: a pricing page redesign in Layer 1, a checkout flow change in Layer 2, and a button color test in Layer 3. User 12345 hashes to variation B in Layer 1, control in Layer 2, and variation A in Layer 3. The user profile service stores these three tuples in Redis keyed by user_id. When User 12345 opens the mobile app, the SDK calls the decision API with their authenticated ID, retrieves the cached assignments, and renders the same variations. If they log out and later log in on desktop, the bucketing ID from the cookie is mapped back to user_id 12345, the profile service returns the original assignments, and the experience remains consistent.
Interview question
Why does the platform compute bucketing as hash(user_id + layer_id + salt) modulo 10000 per layer rather than using a single global hash space?
- a.It guarantees statistical independence between concurrent experiments and prevents traffic stealingCorrect
- b.It ensures a consistent user experience by assigning the same variation across all layers
- c.It minimizes computational overhead when evaluating experiments at scale
- d.It allows new experiments to be added without recomputing existing layer assignments
Why? this is the answer
Per-layer hashing isolates experiments so enrollment in one cannot affect another, preventing collisions and interaction effects. A single global hash would create traffic overlap, and forcing the same variation across layers would destroy statistical independence rather than improve consistency.
Just read this? Test yourself on what you have been reading.
Read the original → docs.developers.optimizely.com
- #experimentation
- #bucketing
- #growth
- #system design
- #ab testing
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on experimentation — each one lists the topics its interview covers.
See open roles