The Incident Commander: Direct the Response, Don't Debug

WHY IT EXISTS: In a crisis, the default response is often disorganized. Multiple engineers might try to fix a problem independently, leading to conflicting actions, duplicated effort, and poor communication with leadership. This ad hoc response often makes an outage worse and last longer. The Incident Commander role exists to impose order on this chaos.

THE MENTAL MODEL: The Incident Commander is an orchestra conductor for an outage. They don't play an instrument (i.e., write code or run commands). Instead, they direct the experts who do. Their job is to maintain a high-level view, delegate tasks to specific owners (like an Operations Lead for hands-on work and a Communications Lead for stakeholder updates), and make critical decisions to guide the team toward service restoration.

HOW IT WORKS: When a significant incident is declared, a designated Incident Commander takes control. They establish a clear communication channel, like a dedicated chat room or video call. The IC's first actions are to understand the impact and start delegating. They assign specific roles to responders, ensuring no one is "just looking." All actions, theories, and communications flow through the IC or their designated leads. This prevents rogue changes and ensures everyone is working from the same set of facts.

WHEN TO USE IT: Use the IC model for any incident that requires coordination between multiple people or teams. If an on-call engineer cannot resolve an issue quickly on their own, or if the impact is spreading, it is time to escalate and appoint an IC. This structure is essential for complex, system-wide failures.

WHEN NOT TO USE IT: The full IC process is overkill for simple, well-understood problems that a single on-call engineer can resolve. For example, a routine server restart for a known flaky service does not require a full incident response. The key is having a clear policy on when an issue graduates to a formal incident.

ONE CANONICAL EXAMPLE: An outage starts in one datacenter. The on-call engineer is overwhelmed as it cascades. Her colleagues start "poking around" without coordinating. A sleepy expert is paged, and another engineer pushes a "fix" to production without telling anyone, causing the remaining servers to die. An IC would have prevented this by establishing a war room, assigning an Operations Lead to investigate, tasking someone else with communications, and explicitly forbidding any unapproved changes.