How would you design an A/B test for two live ML models?

WHAT THIS TESTS: This question probes your ability to design controlled experiments in a live ML serving stack. Interviewers want to see that you understand traffic isolation, consistent user assignment, feature parity, and statistical validity rather than just deploying two endpoints and eyeballing accuracy.

A GOOD ANSWER COVERS: First, deterministic traffic splitting using a hash of a stable user identifier modded into buckets, ensuring the same user always hits the same model variant for the duration of the experiment. Second, pipeline isolation: each model version should consume its own feature store or transformation logic so that preprocessing changes do not confound the comparison. Third, guardrail and success metrics collected in real time, including business KPIs, model latency, throughput, error rates, and prediction distribution drift. Fourth, a power analysis and pre-registration of the primary metric so the runtime and sample size are fixed before launch, preventing peeking bias. Fifth, a rollout and rollback mechanism, such as a progressive canary within the test group, with automatic shutdown if guardrails breach.

COMMON WRONG ANSWERS: Proposing random or round-robin request routing, which breaks user session consistency and inflates variance. Sharing a single feature cache between variants when one model requires different transformations, causing treatment leakage. Comparing offline validation accuracy to online A/B results without accounting for label delay. Running the test until one variant looks better rather than using a fixed horizon, which invites false positives via optional stopping.

LIKELY FOLLOW-UPS: How do you handle delayed feedback labels when evaluating model uplift? What happens if one model has significantly higher latency and users in that bucket bounce more? How would you run an A/B test when models are embedded inside a multi-stage ranking pipeline? How do you detect data distribution shift between the control and treatment groups during the experiment?

ONE CONCRETE EXAMPLE: Imagine a recommendation system testing a new deep learning model against a logistic regression baseline. You hash user_id to 100 buckets, assigning buckets 0-49 to the baseline and 50-99 to the challenger. Each variant reads from a dedicated feature store snapshot with identical data freshness but independent transformation code. You stream event logs into a warehouse where a nightly job computes click-through rate and revenue per user, while a real-time dashboard alerts if p99 latency exceeds 200 milliseconds or if error rate jumps above 0.1 percent. After a pre-calculated two-week run yielding 10 million users per arm, you run a t-test on the primary metric and only then promote the winner.

Source: Google Cloud Architecture Center