Describe the architecture of an A/B testing framework

What's really being asked

This question assesses your understanding of the end-to-end lifecycle of an experiment. It's not just about flipping a coin for users. Interviewers are looking for your ability to design a robust, scalable, and statistically sound system. They want to see if you think about data integrity, user experience consistency, and the potential for misuse or misinterpretation of results by non-technical stakeholders. This is a product-minded system design question.

The full answer

A strong answer will walk through four distinct components. First, user assignment: a deterministic, stateless method like hashing a stable user ID (e.g., hash(userID + experimentID) % 100) to bucket users into variants. This ensures a user consistently sees the same experience. Second, experiment configuration: a centralized service that holds experiment definitions (name, variants, traffic allocation, targeting rules) and serves them to clients. This allows for dynamic start/stop of experiments without code deploys. Third, the data pipeline: clients fire exposure events (user X saw variant A) and conversion events (user X clicked 'buy'). These flow into a data warehouse, where an ETL/ELT job joins them and calculates metrics per variant. Fourth, results presentation: a UI that shows key metrics, the lift (or drop), and crucially, statistical significance measures like p-values and confidence intervals to prevent premature conclusions based on noise.

The mistakes people make

A major red flag is focusing only on the client-side implementation, like if (Math.random() > 0.5). This is not scalable, deterministic, or consistent across sessions/devices. Another common error is neglecting the "exposure event" log; you cannot calculate a conversion rate (conversions / users) if you don't know how many users were in each group. Finally, simply presenting raw conversion counts (e.g., "Variant A got 100 clicks, B got 105") without statistical context is a sign of a junior answer. It ignores sample size and variance.

What usually comes next

Be ready for questions about interaction effects ("What if a user is in two experiments at once?"). The answer involves orthogonal experiment layers or ensuring experiments don't target overlapping user attributes. Another follow-up is about duration: "How long should an experiment run?" The answer is until it reaches a pre-calculated required sample size to achieve statistical power, not just "until the results look good" (p-hacking). They might also ask about handling non-logged-in users (using a device ID or cookie, with caveats).

A concrete example

To test a new checkout button color, we define an experiment with 50% of traffic on "control" (blue) and 50% on "variant" (green). A user with ID 12345 logs in. The client requests experiment configs. The assignment logic computes hash('12345' + 'checkout-color-exp') % 100, which returns 67. Since 67 is > 50, the user is assigned to the "variant" group and sees the green button. The client fires an exposure event: {userID: '12345', experiment: 'checkout-color-exp', variant: 'green'}. If they complete a purchase, a conversion event is fired. After 2 weeks and 500,000 exposed users, the results UI shows the green button has a 4.1% conversion rate vs. blue's 4.0%, with a p-value of 0.04, indicating the result is statistically significant.