Auto-rollback on failed blue-green cutover
Safe cutover design.
shift traffic gradually behind a smart router, use deep health checks plus real SLI monitoring, and auto-revert to blue on breach while blue stays warm.
WHAT THIS TESTS: Your ability to engineer the routing and health layer so a failing green environment is detected fast and rolled back automatically and safely.
A GOOD ANSWER COVERS: Do not flip all traffic at once. Place a smart router, a load balancer or service mesh, in front of both environments and shift traffic in increments, starting small. Health checking must be deep, not a shallow TCP or trivial ping that returns healthy while the app is broken; use a readiness endpoint that exercises real dependencies, and pair it with live SLI monitoring of green's error rate, latency, and saturation. Define automatic rollback triggers: if green's error rate or latency breaches a threshold during the shift, the router immediately drains green and routes one hundred percent back to blue. Critically, keep blue fully warm and running until green is validated, so the rollback is instant and the path always exists.
COMMON WRONG ANSWERS: Relying on shallow ping checks that miss application-level failures, doing an instant full cutover with no incremental gate, or decommissioning blue at cutover, which destroys the rollback path.
LIKELY FOLLOW-UPS: How do you avoid flapping between blue and green? How do you handle in-flight requests during drain? What about connection draining and sticky sessions? How long do you keep blue warm?
ONE CONCRETE EXAMPLE: The router sends ten percent of traffic to green while ninety percent stays on blue. Green's readiness probe exercises its database and cache, and an automated watcher tracks green's error rate against blue's. When green's error rate spikes after taking that ten percent, the watcher trips a threshold and the router drains green and shifts all traffic back to blue within seconds. Because blue never went away and was still serving the majority, users experience minimal disruption, and the team investigates green offline before retrying.
Read the original → docs.aws.amazon.com
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.