Blue/green deployment fails during switch-over with partial decommissioning; recovery and process changes?

WHAT THIS TESTS: This question tests whether you can handle a degraded blue-green deployment when the ideal rollback path is compromised. The interviewer cares about incident command instincts, understanding of the blue-green pattern as described by Martin Fowler, and the ability to balance immediate recovery against long-term process fixes. Specifically they want to see if you recognize that partially decommissioning blue before green is stable violates the core safety mechanism of the pattern.

A GOOD ANSWER COVERS: First, immediate containment: stop shifting traffic, put the application into read-only mode if feasible to flush in-flight transactions, and assess how much of blue is still operational. Second, revive blue: if blue retains any capacity, route traffic back to it immediately and scale it back to full strength while green is drained, because Fowler emphasizes that the fundamental value of this approach is rapid router-level rollback. Third, if blue is truly unusable, pivot to a forward fix or revert on green while accepting the longer recovery window. Fourth, data safety: address missed transactions during the overlap, possibly by feeding transactions to both environments or using database techniques that support both old and new schema versions. Fifth, long-term changes: keep blue fully warm and identical to production until green is bedded down, implement canary or progressive traffic shifting rather than a hard cutover, separate database schema changes from application deployments, and add automated health checks under production-like load before the switch.

COMMON WRONG ANSWERS: A major red flag is saying you would simply roll back to blue without addressing that it has been partially decommissioned. Another weak pattern is suggesting a full rebuild of blue from scratch without estimating the time that would take or the data loss it might cause. Candidates who blame only the bug and ignore the process failure that allowed early decommissioning also score poorly. Ignoring database consistency or acting as if router switches are instantaneous and risk-free indicates shallow familiarity with the pattern.

LIKELY FOLLOW-UPS: The interviewer may ask how you would handle a database schema change during a rollback, or what you would do if the bug causes data corruption in the green environment. They might probe the cost trade-off of keeping two full production environments idle, or ask how long you wait before recycling blue into a staging environment. Another common thread is how to test disaster recovery procedures on every release without impacting users.

ONE CONCRETE EXAMPLE: Imagine you run two identical virtual machine fleets. During the switch, green starts throwing 500 errors under load. You immediately halt the load balancer shift, put the app into read-only mode for thirty seconds to drain writes, then route all traffic back to blue which was scaled down to fifty percent. You scale blue back to one hundred percent while keeping green alive but isolated for debugging. The post-mortem changes the policy so that blue remains at full capacity for at least one hour of stable green metrics before any decommissioning begins, and traffic moves in ten percent increments rather than all at once.

Source: martinfowler.com