Everything in DevOps & Cloud, page 5
Catching rare intermittent failures
Add high-cardinality traces and structured logs with exemplars, use tail-based sampling to keep failing traces, and alert on SLO burn rather than coarse averages.
Measuring on-call health quantitatively
Track pages per shift especially off-hours, actionability rate, time-to-resolve, and load distribution; use trends to prioritize reliability work and protect against burnout.
Designing an escalation policy
Define timeout-based escalation tiers, multiple notification channels, secondary and management layers, coverage across time zones, and severity-based routing.
Anatomy of an effective runbook
Alert meaning and impact, diagnostic steps with dashboard and query links, concrete remediation with rollback, and escalation contacts.
SLO-based alerting and error budgets
Define SLI/SLO, derive an error budget, alert on how fast you burn it using multi-window multi-burn-rate rules so fast burns page and slow burns ticket.
Fixing a noisy non-actionable alert
Review the data to see if it is ever actionable, then tune threshold or duration, re-target at user-facing symptoms, or delete it.
First steps on a p99 latency page
Confirm scope and blast radius, check recent changes and deploys, then look at dependencies and resource saturation.
Symptom-based versus cause-based alerting
Symptom alerts fire on user impact like high error rate or latency; cause alerts fire on internal conditions like high CPU. Page on symptoms, treat causes as diagnostic.
Loki versus Elasticsearch for logs
Loki indexes only labels and stores raw log chunks, cheap but needs label-scoped brute-force search; Elasticsearch full-text indexes content, fast arbitrary search but costly to store…
Writing SLIs in PromQL
Availability is good requests over total using rate and non-5xx counters; latency uses histogram_quantile over rate of buckets summed by le.
Scaling Prometheus for HA and volume
Shard scraping, add long-term object storage and global query via Thanos/Cortex/VictoriaMetrics, run redundant replicas for HA, with dedupe.
Designing a cache health dashboard
Hit ratio, latency, memory and evictions, connections and saturation, with stat panels for current state and time-series for trends.
Correlating logs, metrics, and traces
Propagate a trace and span ID through context, stamp it on logs and metric exemplars, then pivot metric to trace to logs.
Prometheus histogram versus summary
Histograms expose bucket counts and let you compute aggregatable quantiles at query time; summaries compute fixed quantiles per instance that cannot be combined.
High cardinality in time-series databases
Each combination is a separate series consuming memory and index; manage by avoiding unbounded labels and bucketing values.
Core components of Prometheus
Server scrapes targets found via service discovery, stores samples in a local time-series database, with Alertmanager and exporters as helpers.
Histograms versus summaries for latency
Histograms store bucket counts and stay aggregatable with query-time approximate quantiles; summaries precompute quantiles per instance that cannot be averaged.
When to add custom OpenTelemetry instrumentation
Wrap domain logic in custom spans, attach business attributes like tenant or plan, answer revenue and per-customer questions.
Redesigning a high-cardinality request metric
Cardinality is unique label combinations; user_id and raw path are unbounded; redesign by dropping user_id and templating the path.
Trace context and propagation across services
Trace context bundles trace ID, span ID, and flags; propagated via headers like W3C traceparent so each service extracts and continues the trace.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles