Describe the architecture of a generic A/B testing framework

WHAT THIS TESTS: Whether you can architect a system that randomizes users correctly, serves experiments safely at scale, and prevents human misinterpretation of statistics. Interviewers want to see you bridge data science concerns with distributed systems engineering and understand that a dashboard is a user interface that needs safety guardrails.

A GOOD ANSWER COVERS: Four layers in order. First, assignment: use a deterministic hash of a stable user identifier salted with the experiment ID so the same user always sees the same variant, even across devices if logged in. Explain bucketing on a 0 to 1 range and handling traffic allocation changes without re-randomizing existing users. Second, configuration: a centralized experiment config service that returns variant parameters to the application at runtime, decoupling code deploys from experiment changes, with caching and fallback defaults for reliability. Third, metrics and validity: pre-register primary and guardrail metrics before launch, calculate required sample sizes via power analysis, run sample ratio mismatch tests, and use sequential testing or fixed-horizon rules to avoid peeking. Fourth, presentation: dashboards should show point estimates with confidence intervals, statistical significance as a gradient rather than a bright-line p-value, and auto-generated plain-language summaries like variant B is likely better with a lift between 1 and 3 percent.

COMMON WRONG ANSWERS: Randomizing on every page load or API call instead of per user, which destroys experiment validity. Storing experiment state only in local storage or cookies without server-side enforcement. Using p-values as a kill switch without confidence intervals or practical significance thresholds. Building dashboards that let users cherry-pick metrics after the data is collected. Ignoring network or device effects that cause sample ratio mismatch.

LIKELY FOLLOW-UPS: How would you handle overlapping experiments and prevent interaction effects? What happens when a user crosses from mobile web to native app? How do you detect and alert on sample ratio mismatch in real time? How would you scale the config service to millions of requests per second with sub-10 millisecond latency?

ONE CONCRETE EXAMPLE: Imagine a video streaming service testing a new recommendation algorithm. User assignment hashes the account ID and experiment ID into a bucket. The config service returns algorithm version A or B to the API gateway. Pre-registered metrics are watch time per session and subscriber churn. The dashboard shows that variant B has a 2 percent lift in watch time with a 95 percent confidence interval from 0.5 to 3.5 percent, while a guardrail metric shows no increase in churn. The system auto-flags that the result is statistically robust and practically meaningful, so a product manager can ship without calling a data scientist.

Source: Wikipedia: A/B testing