Intermediate concepts in Monitoring & SRE
Embracing Risk: Reliability Is a Budget, Not a Goal
Instead of chasing 100% uptime, SRE treats reliability as a budget. The acceptable downtime, or 'error budget,' is permission to take calculated risks like shipping features faster. The footgun is wasting resources on reliability users won't even notice.
Service Level Indicators: Measuring What Matters
An SLI is a direct measurement of your service's performance, like request latency or error rate. It's the raw data that forms the basis for reliability goals (SLOs). The main footgun is measuring system internals instead of the actual user experience.
Error Budgets: Balancing Reliability and Innovation
An error budget is your service's allowance for unreliability, calculated as 1 minus the SLO. It provides a data-driven signal for when to halt new feature releases and focus on stability, protecting users from repeated SLO misses.
Blameless Postmortems: Fix Systems, Not Blame
A blameless postmortem treats human error as a symptom of a system flaw, not a root cause. It's used after major incidents to find systemic causes and prevent recurrence, assuming everyone acted with good intentions.
The Four Golden Signals of Service Monitoring
The Four Golden Signals provide a simple framework for service health: Latency, Traffic, Errors, and Saturation. Use them for top-level dashboards and alerts. The biggest footgun is ignoring saturation, the leading indicator of future latency issues.
Pull vs. Push: How Your Metrics Get to the Collector
Push vs. Pull metrics is about who starts the conversation. In a push model (like OTLP), the app sends metrics to a collector. In a pull model (like Prometheus), the collector scrapes metrics from the app. The footgun: pull can miss short-lived jobs.
SLO Burn Rate: Alerting on Budget Consumption
SLO burn rate measures how fast you're using your error budget. A burn rate of 2 means you're consuming budget twice as fast as allowed. It's used to trigger alerts for both fast outages and slow-burning problems to prevent SLO breaches.
Error Budget Burndown: Your Service's Reliability Fuel Gauge
An error budget burndown chart is a fuel gauge for your service's reliability. It shows how much 'allowable downtime' you've used, helping you decide when to ship features versus fix bugs. The footgun is ignoring a steep burn rate until it's too late.
OpenTelemetry Collector: The Swiss Army Knife for Telemetry Data
The OpenTelemetry Collector is a universal adapter for your telemetry data. It receives data in one format, processes it, and exports it to multiple backends, letting you centralize configuration and avoid vendor lock-in.
Span: The Building Block of a Distributed Trace
A Span is a single unit of work in a request's journey, like one leg of a flight. Spans capture the duration and context of individual operations (e.g., a DB query), helping you debug latency in distributed systems.
OpenTelemetry Semantic Conventions: A Shared Vocabulary
Think of Semantic Conventions as a universal dictionary for telemetry. They standardize attribute names like http.method across all services, letting you correlate data from different languages, libraries, and platforms seamlessly.
OpenTelemetry Protocol (OTLP): The Universal Adapter for Telemetry
OTLP is the universal adapter for observability, providing a single vendor-neutral protocol for traces, metrics, and logs. Use it to send data from an OpenTelemetry SDK to a collector.
OpenTelemetry Resource: The 'Who' of Your Telemetry
An OpenTelemetry Resource is the return address for your telemetry, describing the service instance that produced it. It attaches stable context like service name and version to every signal, enabling filtering.
Elasticsearch: The Search Engine in the ELK Stack
Elasticsearch is a distributed search engine for querying massive, schema-free JSON datasets via an HTTP API. It's the core of log analysis platforms like the ELK stack, enabling fast search over terabytes of logs.
Downsampling: Trading Granularity for Speed in Time Series Data
Downsampling trades resolution for speed by summarizing old, high-granularity metrics into coarser ones. This makes long-range queries faster and cheaper, common in systems like Thanos for long-term Prometheus data.
Telemetry Correlation: Connecting Dots in Distributed Systems
Telemetry correlation links logs, metrics, and traces from a single user request into one coherent story. It works by propagating a shared context, like a unique ID, across all services involved.
Mean Time to Acknowledge (MTTA): Your First Response Clock
MTTA measures the time from an alert firing to a human acknowledging it. It's about reaction speed, not fix time. On-call teams use this to ensure issues are seen quickly, minimizing downtime.
Mean Time To Repair (MTTR): Measuring Your Fix Velocity
MTTR measures how quickly your team can fix a problem once active work begins. It's the 'wrench time' of incident response, not total outage duration. SREs track it to gauge runbook and diagnostic effectiveness.
The Incident Commander: Direct the Response, Don't Debug
The Incident Commander (IC) is a conductor, not a soloist. During an outage, they coordinate the response, delegate tasks, and manage communication, not fix the bug themselves. This prevents chaos from uncoordinated heroics.
ChatOps: Your CLI Inside Your Chat Room
ChatOps moves your command-line tools into your team's chat, making operations a spectator sport. Instead of a solo SSH session, you run commands via a bot for all to see. Use it for deployments and status checks. The biggest footgun is security.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles