Implement an A/B test for a new checkout flow

What's really being asked

This question tests your understanding of the entire experimentation lifecycle: how to assign users to variants reliably, how to maintain that assignment across different contexts (sessions, devices), and how to structure the data analysis to get a valid result. The interviewer is looking for a systems-level thinker who considers state, data flow, and analytical rigor, not just a developer who can toggle a UI element.

The full answer

A complete answer has three parts. First, describe the technical components: a central bucketing service that takes a user identifier and an experiment name, and deterministically assigns a variant (A or B). This service is called by the application backend or frontend. Second, explain consistent assignment. For logged-in users, hash a stable user ID. This ensures the same user gets the same variant on their phone and desktop. For anonymous users, fall back to a device ID stored in a cookie, but acknowledge the limitation. Third, structure the analytical query. This involves joining an experiment_assignments table (with user_id, experiment_name, variant) with an events table. The query should GROUP BY variant and calculate the conversion rate: COUNT(DISTINCT user_id WHERE event_name = 'purchase_complete') / COUNT(DISTINCT user_id).

The mistakes people make

A major red flag is proposing a purely client-side solution, like using Math.random() and storing the result in a cookie. This is not stable, fails across devices, and is hard to manage. Another weak answer is being hand-wavy about the analysis, e.g., "we'll look at the data." A senior candidate must be able to articulate the specific SQL JOIN and GROUP BY structure needed to get a number. Finally, forgetting to distinguish between logged-in (user ID) and anonymous (device ID) users for bucketing shows a lack of depth.

What usually comes next

How do you handle statistical significance? (Mention p-values, confidence intervals, and needing a minimum sample size/duration). How would you ramp up the experiment from 1% to 50% of users? (The bucketing service configuration should allow for changing the traffic allocation). What if the new checkout flow requires a backend API change? (The API must be backward compatible or versioned, and the bucketing logic should exist on the backend to route requests correctly).

A concrete example

For consistent bucketing, use a hashing algorithm like MurmurHash on a concatenation of the user ID and the experiment name (e.g., hash('user-123:new-checkout-flow-v1')). Then, take the result modulo 100. If the result is <50, assign variant A; otherwise, assign variant B. This is deterministic and provides a uniform distribution. For a 10% rollout, you'd check if the result is <10. The bucketing service exposes an endpoint like GET /assign?userId=123&experiment=checkout-v1 which returns {"variant": "B"}.