Design an A/B Testing Framework

What's really being asked

This question assesses your ability to design a complete, end-to-end system, not just a simple feature. It tests your understanding of distributed systems (config propagation, data pipelines), statistical concepts (hypothesis testing, confidence intervals), and product sense (designing for non-expert users). The interviewer is looking for awareness of the subtle complexities that make A/B testing hard to get right at scale.

The full answer

A strong answer describes four key components in order. First, an Experiment Configuration service where users define hypotheses, metrics, and targeting rules, which generates a unique experiment salt. Second, a User Assignment (bucketing) mechanism, typically a lightweight SDK or service, that uses a deterministic hash like hash(user_id + experiment_salt) % 10000 to assign users to variants consistently. This ensures orthogonality between experiments. Third, a Data Pipeline that logs assignment events and metric events, joining them in a batch or stream processing job (e.g., using Spark or Flink) to attribute user actions to experiment variants. Fourth, a Results & Presentation layer that runs statistical calculations (t-tests, chi-squared tests) to generate p-values and confidence intervals, not just lift percentages. The UI should abstract this complexity, showing clear "winner/loser" status with confidence levels and warning against peeking before the pre-calculated sample size is reached.

The mistakes people make

A frequent red flag is proposing a naive assignment like rand(), which is not sticky for users across sessions. Another is simply calculating metric_count / user_count and calling it "conversion rate" without any statistical rigor; this demonstrates a lack of depth. A critical error is ignoring the "peeking problem"—encouraging users to check results continuously and stop the test when significant, which invalidates the statistics. Finally, failing to mention how to handle multiple overlapping experiments (e.g., through layers or ensuring hash-based assignments are orthogonal) is a miss for a senior role.

What usually comes next

How do you ensure experiment assignments don't interact with each other? (Answer: Orthogonal experiments via independent hashing on different layers/domains). How do you handle long-running experiments or metrics that take days to manifest, like user retention? (Answer: The data pipeline must handle late-arriving data and use appropriate time windows). What is Sample Ratio Mismatch (SRM) and how would you detect it? (Answer: When traffic allocation doesn't match observed data, e.g., a 50/50 split shows 45/55. Use a chi-squared test on assignment counts to detect it, as it often indicates a bug).

A concrete example

To assign 10% of users to an experiment with a 50/50 split, we can use a hash function. Let's say hash(user_id + "my_experiment_salt") produces a uniform integer from 0 to 9999. The experiment runs on users where the hash result is < 1000 (10% of traffic). Within that group, users with a hash result < 500 get the control variant, and users from 500 to 999 get the treatment variant. This is deterministic and scalable. The assignment SDK would log an event like {"event": "assignment", "user_id": "123", "experiment": "new_checkout", "variant": "treatment"} to a Kafka topic for the data pipeline to consume.