Zero-downtime model updates with blue-green or canary

WHAT THIS TESTS This checks whether you can deploy a new model safely, distinguishing the two strategies and pairing them with the monitoring that catches not just crashes but quality regressions unique to ML.

A GOOD ANSWER COVERS In blue-green, you stand up the new model, the green environment, fully alongside the running blue one. You validate green with smoke tests and shadow or mirrored traffic, then flip the load balancer to send all traffic to green at once, keeping blue idle but ready so you can revert instantly if something fails. In canary, you route a small slice, say five percent, of live traffic to the new model while the rest stays on the old one, then gradually increase the slice as metrics hold, which limits blast radius and surfaces problems on a fraction of users. For both, the monitoring must span two layers: infrastructure health such as error rate, latency percentiles, and resource use, and model-quality signals such as the prediction score distribution, drift versus the old model's outputs, and accuracy or precision once ground-truth labels arrive. Define automated rollback triggers tied to these thresholds so a bad model is pulled without a human in the loop.

COMMON WRONG ANSWERS Monitoring only system metrics and missing that a model can be healthy yet produce worse predictions. Having no rollback plan or keeping no warm previous version. Confusing canary with blue-green or sending full traffic to an unvalidated model.

LIKELY FOLLOW-UPS How do you compare quality when labels are delayed? What is shadow deployment and how does it differ? How do you choose canary ramp steps?

ONE CONCRETE EXAMPLE You canary a new recommendation model to five percent of users. Latency stays flat but the new model's click-through is two points lower and its score distribution skews high, signaling miscalibration. Your automated guardrail trips at the CTR threshold and rolls all traffic back to the old model within minutes, before a full rollout could hurt revenue.