A/B test two fraud models in production

WHAT THIS TESTS: Whether you can design an online experiment for a model where errors are costly and labels arrive late, and whether you account for class imbalance and operational safety rather than treating it like a UI button test.

A GOOD ANSWER COVERS: Routing: assign traffic by a stable key such as account or transaction id using consistent hashing, so the same entity always sees the same model and assignment is reproducible. For a sensitive fraud system, consider running the challenger in shadow mode first, scoring live traffic in parallel without taking action, to compare behavior risk-free before granting it real decisions. Metrics: log each model's predictions plus the eventual ground-truth fraud outcome, which typically arrives with a delay from chargebacks or investigations, so the evaluation must wait for labels to mature. Track precision and recall on the rare fraud class, dollars of fraud caught versus missed, and operational guardrails like false-positive rate, which translates to legitimate customers blocked. Deciding the winner: use statistical significance on the primary business metric, net fraud loss, while respecting guardrails, and account for label delay before calling it.

COMMON WRONG ANSWERS: Using overall accuracy on a heavily imbalanced problem, where predicting all-legitimate scores high yet catches nothing. Ignoring label delay and concluding too early. Routing per-request so one account flips between models.

LIKELY FOLLOW-UPS: How do you handle the delay between a prediction and the confirmed fraud label? Why shadow mode before live? How do you balance catching fraud against customer friction?

ONE CONCRETE EXAMPLE: The team hashes account id to send ninety percent to the incumbent and ten percent to the challenger, first in shadow for a week, then live. After labels mature, the challenger shows higher recall and lower net fraud loss with no rise in legitimate-customer blocks, so it is promoted.