Automated rollback for a failed model deploy

WHAT THIS TESTS Whether you can design automated, reliable recovery to a known-good state rather than improvising under an incident.

A GOOD ANSWER COVERS The foundation is that registration does not equal release: the previous model keeps serving until the new version proves healthy. Use a deployment pattern that preserves the old version, blue-green where the old environment stays warm, or canary where only a small traffic slice hits the new model first. Define automated success criteria: health-check endpoints, error rates, latency, and key prediction-quality or business metrics over the canary window. If any criterion fails, the rollback triggers automatically.

TECHNICAL STEPS Detect failure through probes and metric thresholds. Reroute traffic back to the last known-good model version, by flipping the load balancer or endpoint to the blue environment or setting the canary weight to zero. Confirm the restored version is healthy. In the model registry, mark the new version as failed or rejected but do not delete it, preserving lineage for debugging. Emit alerts and a deployment event so on-call and the pipeline both know. Ensure the rollback is idempotent so it is safe to retry.

COMMON WRONG ANSWERS Rolling forward with a quick fix as the primary plan, deleting the failed model and losing lineage, or requiring a human to manually swap models, which is slow and error-prone.

LIKELY FOLLOW-UPS How long to bake a canary, how to handle stateful features during rollback, and how to test the rollback path itself.

ONE CONCRETE EXAMPLE A canary sends five percent of traffic to model version forty-three. Automated monitoring detects error rate spiking above the threshold within minutes; the system sets the canary weight to zero, returning all traffic to version forty-two, marks forty-three as failed in the registry, and pages the team, all without manual action.