Intermediate concepts in Monitoring & SRE, page 3
Latency Injection: Testing How Your System Handles Slowdown
Latency injection intentionally slows down parts of your system to test its resilience. Use it to validate timeout configurations and circuit breakers before a real network lag causes a cascading failure. The footgun is injecting unrealistic delays.
Resource Exhaustion: Starving a System to Death
A resource exhaustion attack is a denial-of-service tactic that starves a system by consuming all of a finite resource like CPU or database connections. It exploits missing rate limits or quotas.
Chaos Mesh: Orchestrated Failure for Resilient Systems
Chaos Mesh is a disaster-as-a-service for Kubernetes. It lets you inject failures like network latency or pod kills to test system resilience. Use it to find weaknesses before they impact users, but be warned: its job is to break things, so scope experiments…
AWS Fault Injection Simulator
AWS Fault Injection Simulator is a controlled chaos button: it breaks resources on purpose to prove your failover works before real disasters. Run it before peak traffic to validate auto-healing.
SRE Team Topologies: Organizing for Fast Flow
Team Topologies structure teams to speed up value delivery by organizing around the software itself. This model helps scale product organizations, adopt cloud-native architectures, or build a platform engineering function.
Shared Ownership Model: Bridging the Dev/Ops Divide
The shared ownership model ends the tug-of-war between developers wanting to ship and operations teams wanting stability. Both teams share responsibility for service quality, using SLOs as a common language.
Capacity Planning: Don't Run Out of Room
Capacity planning matches your system's resources to user demand, crucial for handling traffic spikes or budgeting cloud spend. The main footgun is planning for theoretical 'design capacity' instead of realistic 'effective capacity' which accounts for…
Service Maps: A Live Blueprint of Your Architecture
A service map is a live blueprint of your distributed system, generated from telemetry data. It visualizes service dependencies and health, letting you instantly see an incident's blast radius instead of hunting through Slack for tribal knowledge.
Time-Series Compression: Storing More with Less
Time-series compression stores data more efficiently by saving the *difference* between consecutive points, not the full values. It's key for managing terabyte-scale monitoring and IoT data, often saving over 90% on storage.
Escalation Policy: When to Stop Shipping and Start Fixing
An escalation policy is a pre-agreed plan for when to divert engineers from feature work to fix reliability. When a service's error budget burns too fast, the policy's thresholds trigger specific actions. The footgun is thinking a quick rollback is enough.
Post-Incident Review: Learning from Failure, Blamelessly
A Post-Incident Review (PIR) is a blameless process to learn from an outage, not to assign blame. Use it after a production incident to identify systemic flaws and create action items to prevent repeats.
Load Balancing Algorithms: How to Pick a Server
Load balancing algorithms are the rules a client uses to pick one server from a pool of identical backends. They're used by web proxies routing user traffic and by microservices calling each other.
DevOps Is Culture, SRE Is Engineering
DevOps is a cultural philosophy for fast, safe delivery; SRE is the engineering discipline that implements it with error budgets and SLOs. They are complementary, not rival job titles. The footgun is hiring SREs and declaring DevOps done.
Observability vs Monitoring
Monitoring watches predefined metrics and alerts on known failure modes you anticipated. Observability is the property of being able to ask new questions about a system's internal state from its outputs, letting you debug unknown failures you never predicted…
Metrics in Observability
Metrics are numeric measurements aggregated over time, like counters, gauges, and histograms. They are cheap to store and fast to query, making them ideal for dashboards and alerting, but their pre-aggregation discards per-event detail needed for deep…
SLOs Tied to User Journeys, Not APIs
A user-journey SLO measures the full flow a person experiences, not one microservice's health. If checkout is 99.9% up but payments fail, the metric lied. Teams drown in green per-service dashboards while users are furious.
Auto-Instrumentation
Auto-instrumentation automatically adds telemetry to an application without manual code changes, by hooking into libraries, frameworks, or the runtime. It gives broad baseline observability fast, but produces generic spans that often need manual…
Baggage in Distributed Tracing
Baggage is key-value context propagated alongside a trace across service boundaries, so downstream services can read values set upstream. It enables cross-cutting context like tenant id, but it travels in headers on every hop, so overuse adds latency and leak…
Incident Timeline
An incident timeline is a chronological, timestamped record of what happened during an incident: detection, key events, actions taken, and resolution. It anchors the postmortem in facts, separates symptoms from causes, and reveals detection and response…
Runbooks
A runbook is a documented, step-by-step procedure for handling a specific operational task or known failure, such as responding to an alert. It captures expert knowledge so any on-call engineer can act quickly and consistently, reducing reliance on tribal…
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles