Incident Command: Who Does What in a Crisis
The Incident Command System (ICS) is a playbook for major outages, assigning clear roles to avoid chaos. It's like an emergency response crew for your software. Use it when multiple teams must coordinate.
WHY IT EXISTS: Large-scale incidents are confusing and chaotic. Without a plan, teams waste critical time figuring out who is in charge and how to communicate. A pre-defined response structure allows responders to focus on fixing the problem, not on managing the response itself.
THE MENTAL MODEL: Think of incident response like a fire department's chain of command. When a fire breaks out, everyone has a pre-assigned role—the chief coordinates, others handle the hoses or ladders. The Incident Command System (ICS) applies this to software outages, creating an organizational chart on the fly with an Incident Commander (the chief), an Operations Lead (hands-on fixing), and a Communications Lead (managing updates).
HOW IT WORKS: The framework's goals are the "3Cs": Coordinate, Communicate, and Control. To achieve this, it defines clear roles. The Incident Commander (IC) leads the overall response, directs the high-level state, and delegates tasks. The Operations Lead (OL) directs the hands-on debugging and mitigation efforts. The Communications Lead (CL) manages all communication with stakeholders, both internal and external. The OL and CL report to the IC, creating a clear hierarchy. By default, the person who declares the incident often steps into the IC role initially.
WHEN TO USE IT: Use a formal incident response structure for urgent problems that require multiple people or teams to resolve. The goal is to mitigate impact and restore service when the problem is too large or complex for a single on-call engineer to handle alone. Declaring an incident "early and often" is a common best practice.
WHEN NOT TO USE IT: This formal structure is overkill for minor issues that a single on-call engineer can diagnose and fix quickly. Applying the full command system to a simple bug fix would introduce unnecessary overhead and slow down resolution.
ONE CANONICAL EXAMPLE: The Incident Command System (ICS) was first established by firefighters in 1968 to manage wildfires. Tech companies like Google (with its IMAG framework) and PagerDuty have adapted ICS to manage large-scale system failures, demonstrating its effectiveness beyond physical emergencies.
Read the original → sre.google
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.