Compare Canary and Blue/Green ML deployments and model-specific metrics

What's really being asked

This question tests whether you understand that machine learning deployments have failure modes traditional software does not. A model can return HTTP 200 responses with entirely wrong predictions, so infrastructure metrics are insufficient. The interviewer wants to see that you distinguish deployment mechanics from model validation in production and that you know how to mitigate risk when rolling out models that degrade silently.

The full answer

First, contrast the mechanics. Blue/Green deployments run two identical environments and switch traffic instantly via load balancer or DNS flip, allowing immediate rollback but exposing all users to a bad model at once. Canary releases route a small percentage of traffic, often five or ten percent, to the new model while the baseline handles the rest, then scale up based on metrics. Second, explain why Canary is safer for ML. Because models suffer from silent failures, data drift, and performance regression, you need real-world validation on a subset before full commitment. Third, list ML-specific metrics you would monitor. These include prediction accuracy or business metric lift on the canary slice, prediction distribution compared to baseline using statistical tests like KL divergence or PSI, feature drift to detect training-serving skew, calibration error for probabilistic models, and per-class metrics if dealing with imbalanced data. Fourth, note that standard web service monitoring like latency, throughput, error rate, and CPU or memory usage is necessary but not sufficient for model quality.

The mistakes people make

A major red flag is claiming Canary and Blue/Green are the same except for speed or saying you would monitor only latency and errors. Another mistake is ignoring the need for a holdout or shadow traffic comparison, or suggesting you can validate everything in offline tests before launch. Some candidates also confuse Canary with A/B testing; while related, Canary is about safe rollout and rollback, not necessarily long-term business metric experimentation.

What usually comes next

The interviewer may ask how you would automate rollback when a canary metric degrades, perhaps using Argo Rollouts or Flagger with Prometheus alerts. They might ask how to handle cold start latency when the canary model has a different resource profile, or how you would implement shadow mode before Canary to collect baseline metrics without user impact. Another follow-up is how you compare models when ground truth labels arrive with delay, requiring proxy metrics or delayed feedback loops.

A concrete example

Imagine a fraud detection model update. In Blue/Green, you flip all traffic to the new model at midnight; if it misclassifies fifty percent of fraud as legitimate due to feature drift, you lose money until you flip back. In Canary, you send five percent of checkout traffic to the new model. You monitor the fraud rate on that slice versus the baseline, the distribution of predicted risk scores, and drift in the transaction amount feature. If the canary shows a ten percent drop in recall for high-value fraud within thirty minutes, an automated rollout controller reverts traffic to zero percent for the canary and pages the team.