Skip to content
tezvyn:

All bites

The whole library, newest first. Filter by what you are here for, or pick a topic if you already know.

4330 bites

Page 206

Monitoring & SRE1 min read

Histograms versus summaries for latency

Histograms store bucket counts and stay aggregatable with query-time approximate quantiles; summaries precompute quantiles per instance that cannot be averaged.

Monitoring & SRE1 min read

Core components of Prometheus

Server scrapes targets found via service discovery, stores samples in a local time-series database, with Alertmanager and exporters as helpers.

Monitoring & SRE1 min read

High cardinality in time-series databases

Each combination is a separate series consuming memory and index; manage by avoiding unbounded labels and bucketing values.

Monitoring & SRE1 min read

Prometheus histogram versus summary

Histograms expose bucket counts and let you compute aggregatable quantiles at query time; summaries compute fixed quantiles per instance that cannot be combined.

Monitoring & SRE1 min read

Correlating logs, metrics, and traces

Propagate a trace and span ID through context, stamp it on logs and metric exemplars, then pivot metric to trace to logs.

Monitoring & SRE1 min read

Designing a cache health dashboard

Hit ratio, latency, memory and evictions, connections and saturation, with stat panels for current state and time-series for trends.

Monitoring & SRE1 min read

Scaling Prometheus for HA and volume

Shard scraping, add long-term object storage and global query via Thanos/Cortex/VictoriaMetrics, run redundant replicas for HA, with dedupe.

Monitoring & SRE1 min read

Writing SLIs in PromQL

Availability is good requests over total using rate and non-5xx counters; latency uses histogram_quantile over rate of buckets summed by le.

Monitoring & SRE1 min read

Loki versus Elasticsearch for logs

Loki indexes only labels and stores raw log chunks, cheap but needs label-scoped brute-force search; Elasticsearch full-text indexes content, fast arbitrary search but costly to store…

Monitoring & SRE1 min read

Symptom-based versus cause-based alerting

Symptom alerts fire on user impact like high error rate or latency; cause alerts fire on internal conditions like high CPU. Page on symptoms, treat causes as diagnostic.

Monitoring & SRE1 min read

First steps on a p99 latency page

Confirm scope and blast radius, check recent changes and deploys, then look at dependencies and resource saturation.

Monitoring & SRE1 min read

Fixing a noisy non-actionable alert

Review the data to see if it is ever actionable, then tune threshold or duration, re-target at user-facing symptoms, or delete it.

Monitoring & SRE2 min read

SLO-based alerting and error budgets

Define SLI/SLO, derive an error budget, alert on how fast you burn it using multi-window multi-burn-rate rules so fast burns page and slow burns ticket.

Monitoring & SRE1 min read

Anatomy of an effective runbook

Alert meaning and impact, diagnostic steps with dashboard and query links, concrete remediation with rollback, and escalation contacts.

Monitoring & SRE1 min read

Designing an escalation policy

Define timeout-based escalation tiers, multiple notification channels, secondary and management layers, coverage across time zones, and severity-based routing.

Monitoring & SRE1 min read

Measuring on-call health quantitatively

Track pages per shift especially off-hours, actionability rate, time-to-resolve, and load distribution; use trends to prioritize reliability work and protect against burnout.

Monitoring & SRE2 min read

Catching rare intermittent failures

Add high-cardinality traces and structured logs with exemplars, use tail-based sampling to keep failing traces, and alert on SLO burn rather than coarse averages.

Monitoring & SRE1 min read

Alert silencing versus alert inhibition

Silencing mutes by matcher for a window, inhibition suppresses lower alerts when a higher one fires, dependency-aware.

Monitoring & SRE1 min read

Triaging a 5xx error spike

Confirm impact and scope, declare and assign roles, check recent changes, stop the bleeding before root cause.

Monitoring & SRE1 min read

Primary goal and sections of a post-mortem

Goal is organizational learning and prevention, not blame; include summary, timeline, impact, root cause, action items.