Design a robust automated testing strategy for ML models before production

WHAT THIS TESTS: The interviewer wants to see if you understand that machine learning correctness is distributed, not absolute. They are looking for a multi-layered validation strategy that accounts for data drift, model decay, serving skew, and business impact rather than a single accuracy number. Senior candidates should demonstrate familiarity with statistical testing, progressive delivery, and observability patterns specific to ML pipelines.

A GOOD ANSWER COVERS: Four layers in order. First, offline model validation beyond a single accuracy score using confidence intervals, per-slice performance analysis, and bias detection on held-out test sets that mirror production distributions. Second, data validation and schema enforcement with tools that catch training-serving skew, missing features, and distribution drift before the model ever sees live traffic. Third, shadow or canary deployment where the new model runs against real production traffic but its predictions are not served to users, enabling statistical comparison of latency, throughput, and prediction distributions against the incumbent model. Fourth, gated promotion with automatic rollback triggers based on online business metrics like conversion rate or error rate rather than just model accuracy, because the business metric is what actually matters.

COMMON WRONG ANSWERS: Treating model accuracy on a static test set as the only promotion gate. Ignoring data drift and assuming yesterday's training distribution holds today. Failing to mention training-serving skew or feature store consistency. Proposing A-B testing without first running a shadow phase, which risks user-facing regressions. Suggesting manual QA or human-in-the-loop approval as the primary safety mechanism instead of automated statistical checks.

LIKELY FOLLOW-UPS: How do you detect data drift in high-dimensional feature spaces without excessive false positives? What metrics would you monitor if the model is a ranking system rather than a classifier? How do you validate generative models where there is no ground-truth label? Describe how you would handle a rollback when the model is baked into an edge device or mobile app.

ONE CONCRETE EXAMPLE: A fraud detection model at a payment company is retrained weekly. Before promotion, it must pass offline thresholds on precision and recall across merchant category slices. Then it enters a shadow phase for 24 hours on 10 percent of live traffic, comparing prediction distributions and latency against the production model. If the KL divergence between prediction distributions exceeds 0.05 or latency p99 increases by more than 20 milliseconds, the pipeline halts. Upon passing shadow, the model moves to a 5 percent canary for two hours, gating on a drop in blocked-fraud rate or an increase in false-positive complaints. Any regression triggers an automatic rollback to the previous model version and pages the on-call.