Describe the architecture of a generic A/B testing framework
system design with statistical safety.
hash-based user bucketing, config service, pre-registered metrics, and confidence intervals on dashboards.
request-level randomization or skipping power analysis.
What's really being asked
Whether you can architect a system that randomizes users correctly, serves experiments safely at scale, and prevents human misinterpretation of statistics. Interviewers want to see you bridge data science concerns with distributed systems engineering and understand that a dashboard is a user interface that needs safety guardrails.
The full answer
Four layers in order. First, assignment: use a deterministic hash of a stable user identifier salted with the experiment ID so the same user always sees the same variant, even across devices if logged in. Explain bucketing on a 0 to 1 range and handling traffic allocation changes without re-randomizing existing users. Second, configuration: a centralized experiment config service that returns variant parameters to the application at runtime, decoupling code deploys from experiment changes, with caching and fallback defaults for reliability. Third, metrics and validity: pre-register primary and guardrail metrics before launch, calculate required sample sizes via power analysis, run sample ratio mismatch tests, and use sequential testing or fixed-horizon rules to avoid peeking. Fourth, presentation: dashboards should show point estimates with confidence intervals, statistical significance as a gradient rather than a bright-line p-value, and auto-generated plain-language summaries like variant B is likely better with a lift between 1 and 3 percent.
The mistakes people make
Randomizing on every page load or API call instead of per user, which destroys experiment validity. Storing experiment state only in local storage or cookies without server-side enforcement. Using p-values as a kill switch without confidence intervals or practical significance thresholds. Building dashboards that let users cherry-pick metrics after the data is collected. Ignoring network or device effects that cause sample ratio mismatch.
What usually comes next
How would you handle overlapping experiments and prevent interaction effects? What happens when a user crosses from mobile web to native app? How do you detect and alert on sample ratio mismatch in real time? How would you scale the config service to millions of requests per second with sub-10 millisecond latency?
A concrete example
Imagine a video streaming service testing a new recommendation algorithm. User assignment hashes the account ID and experiment ID into a bucket. The config service returns algorithm version A or B to the API gateway. Pre-registered metrics are watch time per session and subscriber churn. The dashboard shows that variant B has a 2 percent lift in watch time with a 95 percent confidence interval from 0.5 to 3.5 percent, while a guardrail metric shows no increase in churn. The system auto-flags that the result is statistically robust and practically meaningful, so a product manager can ship without calling a data scientist.
Interview question
When ramping an experiment from 10% to 50% traffic, how should the assignment layer treat already-bucketed users?
- a.Assign all previously unassigned users to control to preserve the baseline metric
- b.Randomly promote unassigned users on their next request without checking prior bucketing state
- c.Re-hash every user against the new traffic percentage to ensure unbiased distribution
- d.Keep existing assignments fixed and only bucket previously unassigned users into available slotsCorrect
Why? this is the answer
The card emphasizes that the assignment layer must handle traffic allocation changes without re-randomizing existing users to preserve experiment validity. Re-hashing would change variants mid-experiment and destroy causal inference.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles