Designing an in-house A/B testing platform
System design plus statistical rigor.
Config service, deterministic bucketing SDK, metrics pipeline, analysis engine; fix peeking with fixed samples or sequential tests.
What's really being asked
This is a system design question fused with statistics literacy. The interviewer wants components that fit together and an explicit defense against the most common way experiments lie: stopping early when results look good.
The full answer
Four core pieces. A configuration service stores experiments, variants, traffic allocation, and targeting, exposed to teams via UI and API. An assignment service or client SDK performs deterministic bucketing by hashing user ID plus experiment salt, ensuring sticky, consistent variants and emitting an exposure event. An event collection and metrics pipeline ingests exposures and outcome events, deduplicates, and aggregates per variant. An analysis engine joins exposures to outcomes and computes lift with confidence intervals. For trustworthiness, predetermine sample size and run duration from a power calculation, and do not declare significance before reaching it. Combat the peeking problem, where repeatedly checking a fixed-horizon p-value inflates false positives, by either committing to a fixed sample and only reading results at the end, or adopting sequential testing or Bayesian methods designed for continuous monitoring. Add guardrails: sample-ratio-mismatch alarms, AA tests to validate the pipeline, and multiple-comparison corrections.
The mistakes people make
Describing only the assignment logic and skipping analysis integrity. Showing a live significance dashboard with no warning that watching it invites early stopping. Ignoring sample ratio mismatch and AA testing. Treating one significant metric among many as proof without correcting for multiplicity.
What usually comes next
How does sequential testing avoid the peeking penalty. What is a sample ratio mismatch and what causes it. How do you handle overlapping experiments on the same users.
A concrete example
A team launches a test and the dashboard shows significance on day two. With a fixed-horizon design that p-value is unreliable because they peeked early. Your platform either hides the verdict until the predetermined sample is reached, or uses a sequential test with always-valid bounds so monitoring is safe. Meanwhile an SRM check flags that the control received fifty-five percent of traffic instead of fifty, revealing a bucketing bug that would have invalidated the result regardless.
Interview question
Why does repeatedly checking a fixed-horizon A/B test's p-value and stopping when significant inflate false positives?
- a.P-values become negative when checked too often
- b.It reduces the sample size below the required minimum
- c.It only affects Bayesian methods, not frequentist tests
- d.Each look is another chance for random noise to cross the threshold, so the true error rate exceeds the nominal 5%Correct
Why? this is the answer
Multiple peeks multiply the opportunities for noise to appear significant, raising the actual false-positive rate above the stated level; sequential or fixed-horizon designs address this.
Just read this? Test yourself on what you have been reading.
- #ab-testing
- #system-design
- #statistics
- #peeking-problem
- #experimentation
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on ab-testing — each one lists the topics its interview covers.
See open roles