Fast rollback versus targeted fix in an incident

WHAT THIS TESTS Whether you reason about recovery time, reversibility, and blast radius rather than chasing the most elegant fix. The bias in incident response is toward stopping user pain fast.

A GOOD ANSWER COVERS A full rollback usually gives the lowest and most predictable time-to-recovery, but it reverts everything in that release, can conflict with already-applied database migrations, and may itself be untested as a path. A targeted forward fix is surgical and avoids reverting good changes, but it is slower, riskier to author under stress, and may need its own review and deploy. The choice hinges on data: how confident are you the latest deploy caused it, are there irreversible schema or data migrations in the release, how severe and growing is current impact, and is the rollback path safe and exercised. When in doubt and impact is high, restore service first.

COMMON WRONG ANSWERS Always forward-fixing because rollback feels like defeat, while users suffer. Always rolling back without checking for irreversible migrations.

LIKELY FOLLOW-UPS What if the bad deploy included a forward-only migration? How do you make rollbacks safe by default? When do you accept a partial mitigation like feature-flagging?

ONE CONCRETE EXAMPLE A release at 09:00 causes a 20 percent error rate. The diff is large and the cause unclear, but the release contained no database migration and rollback is automated and tested. You roll back, recovery in three minutes. In a contrasting case, the release applied an irreversible column drop, so rollback would orphan writes; instead you feature-flag off the broken path as a fast partial mitigation while authoring a forward fix, accepting slightly degraded function rather than data loss.