Implement an A/B test for a new checkout flow

What's really being asked

This question assesses your end-to-end systems thinking for experimentation. It's not just about a feature flag. The interviewer wants to see if you can design a system that reliably splits traffic, maintains user consistency, logs events correctly, and enables statistically sound analysis. They're testing your ability to connect product changes to business metrics in a rigorous way.

The full answer

A good answer has three main parts. First, the technical components: a feature flagging service (either in-house or a vendor like LaunchDarkly) to assign users to control (A) or test (B) variants. This service is the source of truth for which user sees what. Second, ensuring consistency: this is achieved by hashing a stable, unique identifier (like a database user_id, not a session ID) to deterministically assign a user to a bucket. This ensures they see the same variant on their phone and laptop. For anonymous users, a device ID or a long-lived cookie can be used, with the caveat that it doesn't solve cross-device consistency. Third, the analytical query: a SQL-like query that joins the experiment assignment events with the purchase_complete events. The query must group by variant (control, test) and calculate the conversion rate for each (e.g., COUNT(DISTINCT user_id WHERE event = 'purchase_complete') / COUNT(DISTINCT user_id WHERE event = 'experiment_assigned')).

The mistakes people make

A major red flag is suggesting a simple client-side randomizer (like Math.random()) without any persistence. This fails the consistency requirement, as the user gets a different experience on every page load or new session, invalidating the test. Another mistake is using a volatile identifier like a session ID for bucketing, which fails cross-session and cross-device consistency. Finally, describing a vague analytical approach without mentioning specific metrics (conversion rate) or the need to group by variant is a weak signal.

What usually comes next

Be ready for "How do you handle statistical significance? What's a p-value?" or "How long would you run this test for? What sample size do you need?". Another common one is "What if the new checkout flow has a bug? How would you monitor for negative impacts and roll back?". They might also ask about handling logged-out vs. logged-in users and the challenges that presents for consistency.

A concrete example

To ensure consistency for a logged-in user, we'd use their stable user ID. For user id: 12345, we'd compute a hash, for example sha256('checkout-flow-v2' + '12345'). We then take the first few bytes of the hash, convert to an integer, and use a modulo operator to bucket the user. If the result modulo 100 is 0-49, they get control; 50-99, they get the test variant. This is deterministic. The analytics query would look like: SELECT variant, COUNT(DISTINCT user_id) AS total_users, COUNT(DISTINCT CASE WHEN event_name = 'purchase_complete' THEN user_id END) AS converted_users FROM experiment_events GROUP BY variant;.