How would you design an A/B test for two live ML models?
Tests production experimentation rigor beyond random splitting. Strong answers cover: consistent user hashing for sticky assignment, isolated feature stores, guardrail metrics, and pre-calculated statistical power.
What's really being asked
This question probes your ability to design controlled experiments in a live ML serving stack. Interviewers want to see that you understand traffic isolation, consistent user assignment, feature parity, and statistical validity rather than just deploying two endpoints and eyeballing accuracy.
The full answer
First, deterministic traffic splitting using a hash of a stable user identifier modded into buckets, ensuring the same user always hits the same model variant for the duration of the experiment. Second, pipeline isolation: each model version should consume its own feature store or transformation logic so that preprocessing changes do not confound the comparison. Third, guardrail and success metrics collected in real time, including business KPIs, model latency, throughput, error rates, and prediction distribution drift. Fourth, a power analysis and pre-registration of the primary metric so the runtime and sample size are fixed before launch, preventing peeking bias. Fifth, a rollout and rollback mechanism, such as a progressive canary within the test group, with automatic shutdown if guardrails breach.
The mistakes people make
Proposing random or round-robin request routing, which breaks user session consistency and inflates variance. Sharing a single feature cache between variants when one model requires different transformations, causing treatment leakage. Comparing offline validation accuracy to online A/B results without accounting for label delay. Running the test until one variant looks better rather than using a fixed horizon, which invites false positives via optional stopping.
What usually comes next
How do you handle delayed feedback labels when evaluating model uplift? What happens if one model has significantly higher latency and users in that bucket bounce more? How would you run an A/B test when models are embedded inside a multi-stage ranking pipeline? How do you detect data distribution shift between the control and treatment groups during the experiment?
A concrete example
Imagine a recommendation system testing a new deep learning model against a logistic regression baseline. You hash user_id to 100 buckets, assigning buckets 0-49 to the baseline and 50-99 to the challenger. Each variant reads from a dedicated feature store snapshot with identical data freshness but independent transformation code. You stream event logs into a warehouse where a nightly job computes click-through rate and revenue per user, while a real-time dashboard alerts if p99 latency exceeds 200 milliseconds or if error rate jumps above 0.1 percent. After a pre-calculated two-week run yielding 10 million users per arm, you run a t-test on the primary metric and only then promote the winner.
Interview question
When A/B testing two live ML models that use different feature transformations, what is the most critical requirement to ensure a valid causal comparison?
- a.Each model variant must use isolated transformation logic and feature stores to prevent preprocessing differences from confounding results.Correct
- b.Both variants should read from the same real-time feature cache to guarantee identical data freshness.
- c.The experiment should run until the challenger shows statistically significant uplift before stopping.
- d.Traffic should be split using round-robin load balancing to ensure equal compute distribution.
Why? this is the answer
Isolating feature stores and transformation logic prevents preprocessing differences from leaking between variants and confounding the experiment. Option B is tempting because identical freshness sounds correct, but sharing caches causes treatment leakage when transformation logic differs.
Just read this? Test yourself on what you have been reading.
Read the original → docs.cloud.google.com
- #mlops
- #ab-testing
- #experimentation
- #production-ml
- #infrastructure
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on mlops — each one lists the topics its interview covers.
See open roles