SRE
205 bites tagged SRE — interview questions with model answers, and 60-second explainers.
Hindsight Bias: The 'Knew-It-All-Along' Postmortem Trap
Hindsight bias makes past failures seem obvious. In postmortems, this leads to blaming engineers for not seeing what's now clear, instead of fixing the system. The footgun is judging past decisions with present knowledge, which hides real systemic flaws.
Swiss Cheese Model: Layered Defenses Against Failure
Think of system defenses as slices of Swiss cheese. An accident happens only when the holes—weaknesses in each layer—align. It's used in post-mortems to see how small failures combine into a major outage.
Beyond 'Root Cause': Proximate vs. Contributing Factors
A proximate cause is an incident's final trigger, while contributing factors are the conditions that made it possible. This helps post-mortems move beyond blame to find systemic risks.
OODA Loop: Winning the Incident Response Race
The OODA loop (Observe, Orient, Decide, Act) is a model for making fast decisions under pressure. During an incident, the team that cycles fastest wins. It's used for triaging alerts and debugging live outages.
5 Whys: Find the Root Cause, Not Just the Symptom
The 5 Whys technique digs past symptoms to find a problem's true root cause. It's used in postmortems to understand system failures by repeatedly asking "Why?". The footgun is stopping too early or blaming people instead of broken processes.
Public Status Page: Your System's Voice During an Outage
A public status page is a dedicated site for communicating your service's health, turning "is it down?" support tickets into a single source of truth. It's used to report outages, degradation, and scheduled maintenance for public-facing services.
Incident Command System (ICS): Taming Outage Chaos
ICS gives a chaotic outage a clear command structure, defining roles so everyone knows who's in charge. It's used for major service outages or security breaches where multiple teams must coordinate. The footgun: not pre-assigning roles before a crisis hits.
The Incident Management Lifecycle
Incident management is a structured loop for handling service disruptions. It's not just about fixing the problem now, but identifying, analyzing, and correcting hazards to prevent them from happening again. The biggest mistake is skipping the 'prevent' step.
Game Days: Practice Breaking Your System Before It Breaks Itself
A Game Day is a live fire drill for your systems. You intentionally inject failure—like shutting down a service—to see how your team and automation respond, finding weaknesses before a real outage does.
Follow-the-Sun: A Global Relay Race for Work
Follow-the-sun is a global relay race for work. Teams hand off tasks to the next timezone as their day ends, enabling 24/7 coverage without requiring overnight shifts. The footgun is a poor handoff, where lost context erases all the time savings.
ChatOps: Your CLI Inside Your Chat Room
ChatOps moves your command-line tools into your team's chat, making operations a spectator sport. Instead of a solo SSH session, you run commands via a bot for all to see. Use it for deployments and status checks. The biggest footgun is security.
The Incident Commander: Direct the Response, Don't Debug
The Incident Commander (IC) is a conductor, not a soloist. During an outage, they coordinate the response, delegate tasks, and manage communication, not fix the bug themselves. This prevents chaos from uncoordinated heroics.
Mean Time To Repair (MTTR): Measuring Your Fix Velocity
MTTR measures how quickly your team can fix a problem once active work begins. It's the 'wrench time' of incident response, not total outage duration. SREs track it to gauge runbook and diagnostic effectiveness.
Mean Time to Acknowledge (MTTA): Your First Response Clock
MTTA measures the time from an alert firing to a human acknowledging it. It's about reaction speed, not fix time. On-call teams use this to ensure issues are seen quickly, minimizing downtime.
On-Call Management Platforms: Who Wakes Up?
An on-call platform is a smart switchboard for production alerts, ensuring the right engineer gets paged when things break. It connects monitoring tools to on-call schedules and escalation rules.
Incident Command: Who Does What in a Crisis
The Incident Command System (ICS) is a playbook for major outages, assigning clear roles to avoid chaos. It's like an emergency response crew for your software. Use it when multiple teams must coordinate.
On-Call Rotations: Engineering Reliability Under Pressure
On-call rotations are the human backstop for service reliability, with engineers responding to alerts in minutes. This is critical for high-availability services like search or email.
Query Federation: Combining Prometheus Servers
Query federation lets one Prometheus server scrape metrics from another, creating a meta-monitor. Use it to build a global view from local servers or to combine application and infrastructure metrics for richer alerts.
Inverted Index: The Engine of Fast Log Search
An inverted index makes log search fast by mapping terms to the logs containing them, like a book's index. It powers platforms like Splunk or Elasticsearch, enabling instant searches across terabytes of data. The footgun is indexing high-cardinality fields.
Loki: The Log System That Indexes Labels, Not Text
Loki is a log system that indexes only metadata labels, not the full log content. This makes it cheaper and simpler to run than full-text indexing systems, storing compressed logs in object storage.
Telemetry Correlation: Connecting Dots in Distributed Systems
Telemetry correlation links logs, metrics, and traces from a single user request into one coherent story. It works by propagating a shared context, like a unique ID, across all services involved.
Downsampling: Trading Granularity for Speed in Time Series Data
Downsampling trades resolution for speed by summarizing old, high-granularity metrics into coarser ones. This makes long-range queries faster and cheaper, common in systems like Thanos for long-term Prometheus data.
Grafana: Your Single Pane of Glass for Observability
Grafana is the universal dashboard for your system's health, visualizing metrics, logs, and traces from different sources in one place. Use it to monitor application performance and infrastructure health.
Prometheus: Monitoring with a Dimensional Data Model
Prometheus models system health as labeled time series, not just flat metrics. It pulls data from targets, making it ideal for dynamic cloud environments like Kubernetes. The footgun is using high-cardinality labels, which can overwhelm its storage.
Get SRE bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.