Concepts in Monitoring & SRE, page 3
Incident Command: Who Does What in a Crisis
The Incident Command System (ICS) is a playbook for major outages, assigning clear roles to avoid chaos. It's like an emergency response crew for your software. Use it when multiple teams must coordinate.
On-Call Management Platforms: Who Wakes Up?
An on-call platform is a smart switchboard for production alerts, ensuring the right engineer gets paged when things break. It connects monitoring tools to on-call schedules and escalation rules.
Mean Time to Acknowledge (MTTA): Your First Response Clock
MTTA measures the time from an alert firing to a human acknowledging it. It's about reaction speed, not fix time. On-call teams use this to ensure issues are seen quickly, minimizing downtime.
Mean Time To Repair (MTTR): Measuring Your Fix Velocity
MTTR measures how quickly your team can fix a problem once active work begins. It's the 'wrench time' of incident response, not total outage duration. SREs track it to gauge runbook and diagnostic effectiveness.
The Incident Commander: Direct the Response, Don't Debug
The Incident Commander (IC) is a conductor, not a soloist. During an outage, they coordinate the response, delegate tasks, and manage communication, not fix the bug themselves. This prevents chaos from uncoordinated heroics.
ChatOps: Your CLI Inside Your Chat Room
ChatOps moves your command-line tools into your team's chat, making operations a spectator sport. Instead of a solo SSH session, you run commands via a bot for all to see. Use it for deployments and status checks. The biggest footgun is security.
Follow-the-Sun: A Global Relay Race for Work
Follow-the-sun is a global relay race for work. Teams hand off tasks to the next timezone as their day ends, enabling 24/7 coverage without requiring overnight shifts. The footgun is a poor handoff, where lost context erases all the time savings.
Game Days: Practice Breaking Your System Before It Breaks Itself
A Game Day is a live fire drill for your systems. You intentionally inject failure—like shutting down a service—to see how your team and automation respond, finding weaknesses before a real outage does.
The Incident Management Lifecycle
Incident management is a structured loop for handling service disruptions. It's not just about fixing the problem now, but identifying, analyzing, and correcting hazards to prevent them from happening again. The biggest mistake is skipping the 'prevent' step.
Incident Command System (ICS): Taming Outage Chaos
ICS gives a chaotic outage a clear command structure, defining roles so everyone knows who's in charge. It's used for major service outages or security breaches where multiple teams must coordinate. The footgun: not pre-assigning roles before a crisis hits.
Public Status Page: Your System's Voice During an Outage
A public status page is a dedicated site for communicating your service's health, turning "is it down?" support tickets into a single source of truth. It's used to report outages, degradation, and scheduled maintenance for public-facing services.
5 Whys: Find the Root Cause, Not Just the Symptom
The 5 Whys technique digs past symptoms to find a problem's true root cause. It's used in postmortems to understand system failures by repeatedly asking "Why?". The footgun is stopping too early or blaming people instead of broken processes.
OODA Loop: Winning the Incident Response Race
The OODA loop (Observe, Orient, Decide, Act) is a model for making fast decisions under pressure. During an incident, the team that cycles fastest wins. It's used for triaging alerts and debugging live outages.
Beyond 'Root Cause': Proximate vs. Contributing Factors
A proximate cause is an incident's final trigger, while contributing factors are the conditions that made it possible. This helps post-mortems move beyond blame to find systemic risks.
Swiss Cheese Model: Layered Defenses Against Failure
Think of system defenses as slices of Swiss cheese. An accident happens only when the holes—weaknesses in each layer—align. It's used in post-mortems to see how small failures combine into a major outage.
Hindsight Bias: The 'Knew-It-All-Along' Postmortem Trap
Hindsight bias makes past failures seem obvious. In postmortems, this leads to blaming engineers for not seeing what's now clear, instead of fixing the system. The footgun is judging past decisions with present knowledge, which hides real systemic flaws.
Declarative vs. Imperative Automation
Declarative automation defines the desired end state (“what”), not the steps to get there (“how”). It's used in tools like Kubernetes to manage complex infrastructure, letting the system figure out the details.
Configuration Management: Enforcing Desired State
Configuration Management treats your system's setup as code to prevent "configuration drift." It's used to reliably provision servers, deploy apps, and manage fleets, ensuring every component matches its intended design and is reproducible.
Executable Runbooks: Code, Not Just Checklists
An executable runbook turns a procedural document into an automated script. Instead of reading steps, you run them. It's used for incident response or maintenance, ensuring consistency. The footgun is not making them idempotent, which can worsen an outage.
Auto-Remediation: Automated Fixes for Common Failures
Auto-remediation is a system's immune response, automatically detecting and fixing known problems like a crashed service. It's a core SRE practice for improving availability, but a bad script can create a 'remediation storm' that worsens an outage.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles