More in Monitoring & SRE — page 5
Measuring on-call health quantitatively
WHAT IT TESTS: turning on-call pain into data that justifies investment. OUTLINE: track pages per shift especially off-hours, actionability rate, time-to-resolve, and load distribution; use trends to prioritize reliability work and protect against burnout.
Designing an escalation policy
WHAT IT TESTS: ensuring critical alerts always reach someone. OUTLINE: define timeout-based escalation tiers, multiple notification channels, secondary and management layers, coverage across time zones, and severity-based routing.
Anatomy of an effective runbook
WHAT IT TESTS: knowing what makes a runbook usable under stress. OUTLINE: alert meaning and impact, diagnostic steps with dashboard and query links, concrete remediation with rollback, and escalation contacts.
SLO-based alerting and error budgets
WHAT IT TESTS: alerting on budget burn rate, not static thresholds. OUTLINE: define SLI/SLO, derive an error budget, alert on how fast you burn it using multi-window multi-burn-rate rules so fast burns page and slow burns ticket.
Fixing a noisy non-actionable alert
WHAT IT TESTS: treating alert noise as a fixable problem, not just muting. OUTLINE: review the data to see if it is ever actionable, then tune threshold or duration, re-target at user-facing symptoms, or delete it.
First steps on a p99 latency page
WHAT IT TESTS: a structured triage instinct, not random poking. OUTLINE: confirm scope and blast radius, check recent changes and deploys, then look at dependencies and resource saturation. RED FLAG: diving into code or restarting things before scoping impact.
Symptom-based versus cause-based alerting
WHAT IT TESTS: understanding why user-facing symptoms make better pages. OUTLINE: symptom alerts fire on user impact like high error rate or latency; cause alerts fire on internal conditions like high CPU. Page on symptoms, treat causes as diagnostic.
Loki versus Elasticsearch for logs
WHAT IT TESTS: how indexing choice drives cost and query behavior. OUTLINE: Loki indexes only labels and stores raw log chunks, cheap but needs label-scoped brute-force search; Elasticsearch full-text indexes content, fast arbitrary search but costly to store…
Writing SLIs in PromQL
WHAT IT TESTS: turning SLOs into correct PromQL. OUTLINE: availability is good requests over total using rate and non-5xx counters; latency uses histogram_quantile over rate of buckets summed by le.
Scaling Prometheus for HA and volume
WHAT IT TESTS: scaling a single Prometheus past its limits. OUTLINE: shard scraping, add long-term object storage and global query via Thanos/Cortex/VictoriaMetrics, run redundant replicas for HA, with dedupe.
Designing a cache health dashboard
WHAT IT TESTS: knowing which cache signals matter and matching them to visualizations. OUTLINE: hit ratio, latency, memory and evictions, connections and saturation, with stat panels for current state and time-series for trends.
Correlating logs, metrics, and traces
WHAT IT TESTS: knowing the trace ID is the glue across signals. OUTLINE: propagate a trace and span ID through context, stamp it on logs and metric exemplars, then pivot metric to trace to logs. RED FLAG: correlating only by timestamp.
Prometheus histogram versus summary
WHAT IT TESTS: where quantiles are computed and whether they aggregate. OUTLINE: histograms expose bucket counts and let you compute aggregatable quantiles at query time; summaries compute fixed quantiles per instance that cannot be combined.
High cardinality in time-series databases
WHAT IT TESTS: knowing cardinality equals number of unique label combinations. OUTLINE: each combination is a separate series consuming memory and index; manage by avoiding unbounded labels and bucketing values.
Core components of Prometheus
WHAT IT TESTS: understanding Prometheus's pull model and TSDB. OUTLINE: server scrapes targets found via service discovery, stores samples in a local time-series database, with Alertmanager and exporters as helpers.
Histograms versus summaries for latency
WHAT IT TESTS: understanding where quantiles are computed and whether they aggregate. OUTLINE: histograms store bucket counts and stay aggregatable with query-time approximate quantiles; summaries precompute quantiles per instance that cannot be averaged.
When to add custom OpenTelemetry instrumentation
WHAT IT TESTS: knowing auto-instrumentation only sees framework boundaries, not business meaning. OUTLINE: wrap domain logic in custom spans, attach business attributes like tenant or plan, answer revenue and per-customer questions.
Redesigning a high-cardinality request metric
WHAT IT TESTS: Diagnosing and fixing label explosion. OUTLINE: Cardinality is unique label combinations; user_id and raw path are unbounded; redesign by dropping user_id and templating the path.
Trace context and propagation across services
WHAT IT TESTS: How trace context carries correlation across services. OUTLINE: Trace context bundles trace ID, span ID, and flags; propagated via headers like W3C traceparent so each service extracts and continues the trace.
OpenTelemetry API, SDK, and Collector
WHAT IT TESTS: Understanding the OTel pipeline. OUTLINE: API defines vendor-neutral instrumentation, SDK implements and exports it, Collector receives, processes, and routes telemetry to backends.