Intermediate everything in Monitoring & SRE, page 6
Rate Limiting: Your API's Bouncer
Rate limiting acts as a bouncer for your API, controlling traffic to protect your service. It's used on public APIs to prevent abuse, ensure fair usage, and defend against denial-of-service attacks.
Graceful Degradation: Failing Better, Not All at Once
Instead of total failure, graceful degradation lets a system shed non-essential features to stay partially available. It's a fault tolerance strategy for when a component fails, ensuring core functions survive. The footgun is not defining what's 'core.'
Performance Budgets: Set Limits to Stay Fast
A performance budget is a hard limit on metrics like bundle size or load time, acting as a guardrail against regressions. It's used in CI/CD to fail builds that exceed size limits or in monitoring to alert when load times degrade.
Time-Series Forecasting: Predicting the Future from the Past
Time-series forecasting uses past data points, ordered by time, to predict future values. It's used for capacity planning and financial modeling. The footgun is assuming past trends will hold, as sudden system changes can invalidate all predictions.
Scalability Testing: Will More Hardware Fix It?
Scalability testing answers 'Will more hardware fix it?' by measuring how performance improves when you add resources. The common footgun is confusing it with load testing, which just finds the breaking point under a given load.
Soak Testing: Finding Bugs That Only Time Reveals
Soak testing is like running a marathon, not a sprint, to find bugs that only time reveals. It applies a typical production load over a long period to uncover slow memory leaks or resource exhaustion. The footgun is confusing it with stress testing.
Quality Gates: Your Automated Release Checklist
A quality gate is an automated checklist that decides if code is ready for release. It runs in your CI pipeline, blocking merges or failing builds if metrics like code coverage or bug counts don't meet predefined standards.
Feature Flags: Ship Code Now, Release It Later
Feature flags are like light switches for your code, letting you turn features on or off in production without a new deployment. This enables canary releases and A/B tests. The main footgun is letting old flags accumulate, creating a maze of dead code.
Continuous Delivery: Ship Reliably, Anytime
Continuous Delivery (CD) makes software releases a boring, routine event. Your codebase is always in a deployable state, allowing you to ship small, incremental updates reliably and at any time. The footgun is confusing it with Continuous Deployment.
Executable Runbooks: Code, Not Just Checklists
An executable runbook turns a procedural document into an automated script. Instead of reading steps, you run them. It's used for incident response or maintenance, ensuring consistency. The footgun is not making them idempotent, which can worsen an outage.
Configuration Management: Enforcing Desired State
Configuration Management treats your system's setup as code to prevent "configuration drift." It's used to reliably provision servers, deploy apps, and manage fleets, ensuring every component matches its intended design and is reproducible.
Hindsight Bias: The 'Knew-It-All-Along' Postmortem Trap
Hindsight bias makes past failures seem obvious. In postmortems, this leads to blaming engineers for not seeing what's now clear, instead of fixing the system. The footgun is judging past decisions with present knowledge, which hides real systemic flaws.
Swiss Cheese Model: Layered Defenses Against Failure
Think of system defenses as slices of Swiss cheese. An accident happens only when the holes—weaknesses in each layer—align. It's used in post-mortems to see how small failures combine into a major outage.
Beyond 'Root Cause': Proximate vs. Contributing Factors
A proximate cause is an incident's final trigger, while contributing factors are the conditions that made it possible. This helps post-mortems move beyond blame to find systemic risks.
5 Whys: Find the Root Cause, Not Just the Symptom
The 5 Whys technique digs past symptoms to find a problem's true root cause. It's used in postmortems to understand system failures by repeatedly asking "Why?". The footgun is stopping too early or blaming people instead of broken processes.
Public Status Page: Your System's Voice During an Outage
A public status page is a dedicated site for communicating your service's health, turning "is it down?" support tickets into a single source of truth. It's used to report outages, degradation, and scheduled maintenance for public-facing services.
Follow-the-Sun: A Global Relay Race for Work
Follow-the-sun is a global relay race for work. Teams hand off tasks to the next timezone as their day ends, enabling 24/7 coverage without requiring overnight shifts. The footgun is a poor handoff, where lost context erases all the time savings.
ChatOps: Your CLI Inside Your Chat Room
ChatOps moves your command-line tools into your team's chat, making operations a spectator sport. Instead of a solo SSH session, you run commands via a bot for all to see. Use it for deployments and status checks. The biggest footgun is security.
The Incident Commander: Direct the Response, Don't Debug
The Incident Commander (IC) is a conductor, not a soloist. During an outage, they coordinate the response, delegate tasks, and manage communication, not fix the bug themselves. This prevents chaos from uncoordinated heroics.
Mean Time To Repair (MTTR): Measuring Your Fix Velocity
MTTR measures how quickly your team can fix a problem once active work begins. It's the 'wrench time' of incident response, not total outage duration. SREs track it to gauge runbook and diagnostic effectiveness.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles