Design a multivariate experimentation platform with collision-free concurrent bucketing and cross-device consistency

WHAT THIS TESTS: This tests your ability to design a growth platform core primitive: assigning users to variations deterministically, in parallel, and persistently. Interviewers care about four things: statistical independence between experiments, cross-device consistency, identity resolution, and operational safety at scale. They want to see you understand that bucketing is not random sampling but a deterministic function of user identity and experiment configuration.

A GOOD ANSWER COVERS: First, orthogonal layers or namespaces. Each experiment lives in a layer identified by a layer ID. The bucketing hash is computed as hash(user_id + layer_id + salt) modulo 10000, which guarantees that Experiment A in Layer 1 and Experiment B in Layer 2 are independent and cannot steal traffic from each other. For multivariate tests, each factor combination is a variation and traffic is allocated across the full factorial grid.

Second, a user profile service. When a user is bucketed, the decision tuple (experiment_id, variation_id, timestamp) is written to a fast lookup store keyed by the resolved user ID. On subsequent requests, the service reads before re-computing the hash, ensuring the user never flips variations. This is how Optimizely implements sticky bucketing.

Third, identity resolution. The system evaluates a hierarchy: authenticated user ID is primary, then a stable bucketing ID from a first-party cookie or device ID, then a probabilistic ID. When a user logs in, the anonymous profile is merged into the authenticated profile, preserving existing assignments and avoiding double-counting.

Fourth, audience segmentation before bucketing. Dynamic segments are evaluated using attributes passed at decision time. If a user qualifies, they enter the experiment; if not, they receive the default and are not counted in the denominator. This prevents dilution of statistical power.

COMMON WRONG ANSWERS: Using a single global hash space for all experiments causes traffic collisions and interaction effects. Relying on random number generators without a seed breaks reproducibility. Storing assignments only in browser localStorage fails cross-device and incognito scenarios. Re-bucketing on every evaluation destroys experiment validity and user trust. Ignoring the merge of anonymous to authenticated profiles inflates user counts and corrupts metrics.

LIKELY FOLLOW-UPS: How do you handle mutual exclusion when two experiments in the same layer must not overlap? How do you implement a global holdout group that sees no experiments? What happens if the user profile service is down: do you fail open or closed? How do you prevent a bot from polluting the control or treatment? How do you support server-side rendering without a flash of wrong variation?

ONE CONCRETE EXAMPLE: Imagine a SaaS product running three experiments: a pricing page redesign in Layer 1, a checkout flow change in Layer 2, and a button color test in Layer 3. User 12345 hashes to variation B in Layer 1, control in Layer 2, and variation A in Layer 3. The user profile service stores these three tuples in Redis keyed by user_id. When User 12345 opens the mobile app, the SDK calls the decision API with their authenticated ID, retrieves the cached assignments, and renders the same variations. If they log out and later log in on desktop, the bucketing ID from the cookie is mapped back to user_id 12345, the profile service returns the original assignments, and the experience remains consistent.

Source: docs.developers.optimizely.com