Designing an in-house A/B testing platform

WHAT THIS TESTS This is a system design question fused with statistics literacy. The interviewer wants components that fit together and an explicit defense against the most common way experiments lie: stopping early when results look good.

A GOOD ANSWER COVERS Four core pieces. A configuration service stores experiments, variants, traffic allocation, and targeting, exposed to teams via UI and API. An assignment service or client SDK performs deterministic bucketing by hashing user ID plus experiment salt, ensuring sticky, consistent variants and emitting an exposure event. An event collection and metrics pipeline ingests exposures and outcome events, deduplicates, and aggregates per variant. An analysis engine joins exposures to outcomes and computes lift with confidence intervals. For trustworthiness, predetermine sample size and run duration from a power calculation, and do not declare significance before reaching it. Combat the peeking problem, where repeatedly checking a fixed-horizon p-value inflates false positives, by either committing to a fixed sample and only reading results at the end, or adopting sequential testing or Bayesian methods designed for continuous monitoring. Add guardrails: sample-ratio-mismatch alarms, AA tests to validate the pipeline, and multiple-comparison corrections.

COMMON WRONG ANSWERS Describing only the assignment logic and skipping analysis integrity. Showing a live significance dashboard with no warning that watching it invites early stopping. Ignoring sample ratio mismatch and AA testing. Treating one significant metric among many as proof without correcting for multiplicity.

LIKELY FOLLOW-UPS How does sequential testing avoid the peeking penalty. What is a sample ratio mismatch and what causes it. How do you handle overlapping experiments on the same users.

ONE CONCRETE EXAMPLE A team launches a test and the dashboard shows significance on day two. With a fixed-horizon design that p-value is unreliable because they peeked early. Your platform either hides the verdict until the predetermined sample is reached, or uses a sequential test with always-valid bounds so monitoring is safe. Meanwhile an SRM check flags that the control received fifty-five percent of traffic instead of fifty, revealing a bucketing bug that would have invalidated the result regardless.