A/B test two fraud models in production
production model experimentation design.
randomize by entity, consider shadow mode first, collect precision/recall and business loss, decide with significance and guardrails.
What's really being asked
Whether you can design an online experiment for a model where errors are costly and labels arrive late, and whether you account for class imbalance and operational safety rather than treating it like a UI button test.
The full answer
Routing: assign traffic by a stable key such as account or transaction id using consistent hashing, so the same entity always sees the same model and assignment is reproducible. For a sensitive fraud system, consider running the challenger in shadow mode first, scoring live traffic in parallel without taking action, to compare behavior risk-free before granting it real decisions. Metrics: log each model's predictions plus the eventual ground-truth fraud outcome, which typically arrives with a delay from chargebacks or investigations, so the evaluation must wait for labels to mature. Track precision and recall on the rare fraud class, dollars of fraud caught versus missed, and operational guardrails like false-positive rate, which translates to legitimate customers blocked. Deciding the winner: use statistical significance on the primary business metric, net fraud loss, while respecting guardrails, and account for label delay before calling it.
The mistakes people make
Using overall accuracy on a heavily imbalanced problem, where predicting all-legitimate scores high yet catches nothing. Ignoring label delay and concluding too early. Routing per-request so one account flips between models.
What usually comes next
How do you handle the delay between a prediction and the confirmed fraud label? Why shadow mode before live? How do you balance catching fraud against customer friction?
A concrete example
The team hashes account id to send ninety percent to the incumbent and ten percent to the challenger, first in shadow for a week, then live. After labels mature, the challenger shows higher recall and lower net fraud loss with no rise in legitimate-customer blocks, so it is promoted.
Interview question
Why is overall accuracy a poor primary metric when A/B testing two fraud detection models?
- a.Accuracy requires both models to share the same threshold
- b.Accuracy ignores the routing key used to split traffic
- c.Fraud is highly imbalanced, so a model that flags nothing still scores high accuracyCorrect
- d.Accuracy cannot be computed in real time
Why? this is the answer
With rare fraud, predicting everything legitimate yields high accuracy while catching no fraud, so precision, recall, and dollar loss matter instead. Accuracy is computable in real time; the issue is imbalance.
Just read this? Test yourself on what you have been reading.
Read the original → wallaroo.ai
- #ab-testing
- #fraud-detection
- #model-deployment
- #shadow-deployment
- #class-imbalance
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on ab-testing — each one lists the topics its interview covers.
See open roles