Advanced everything in Monitoring & SRE, page 2
Alert silencing versus alert inhibition
Silencing mutes by matcher for a window, inhibition suppresses lower alerts when a higher one fires, dependency-aware.
Catching rare intermittent failures
Add high-cardinality traces and structured logs with exemplars, use tail-based sampling to keep failing traces, and alert on SLO burn rather than coarse averages.
Measuring on-call health quantitatively
Track pages per shift especially off-hours, actionability rate, time-to-resolve, and load distribution; use trends to prioritize reliability work and protect against burnout.
Loki versus Elasticsearch for logs
Loki indexes only labels and stores raw log chunks, cheap but needs label-scoped brute-force search; Elasticsearch full-text indexes content, fast arbitrary search but costly to store…
Writing SLIs in PromQL
Availability is good requests over total using rate and non-5xx counters; latency uses histogram_quantile over rate of buckets summed by le.
Scaling Prometheus for HA and volume
Shard scraping, add long-term object storage and global query via Thanos/Cortex/VictoriaMetrics, run redundant replicas for HA, with dedupe.
Histograms versus summaries for latency
Histograms store bucket counts and stay aggregatable with query-time approximate quantiles; summaries precompute quantiles per instance that cannot be averaged.
When to add custom OpenTelemetry instrumentation
Wrap domain logic in custom spans, attach business attributes like tenant or plan, answer revenue and per-customer questions.
Fixing watermelon SLOs that hide unhappy users
Green-outside, red-inside SLOs come from server-side measurement, coarse aggregation, wrong events, and lax thresholds; fix by measuring at the client and using meaningful SLIs.
Launching a risky feature with no error budget left
Bring burn history, what consumed the budget, blast radius, and rollback safety; offer mitigations like flags and canaries.
Setting SLOs across a dependency chain
Serial dependencies multiply, so each backend needs a tighter SLO than the target; mitigate with redundancy, caching, and graceful degradation.
Symptom-based vs cause-based alerting
Symptom alerts fire on user-visible impact, cause alerts on internal conditions; page on symptoms to cut noise and catch unforeseen failures.
Head-based vs tail-based trace sampling
Head-based decides at trace start cheaply but blindly; tail-based waits for the full trace to keep errors and slow requests.
What is high-cardinality data in Prometheus?
Cardinality is the count of unique label combinations; each is a separate series; explosion blows up memory and query cost.
Error budget policy across dependent microservices?
Set per-service SLOs but anchor on user-facing journey SLOs, budget for dependency error via the multiplication of availabilities, attribute downstream-caused failures correctly, and use…
SRE vs traditional ops on a recurring alert?
Traditional ops repeatedly handles the alert manually and scales by adding people; SRE treats it as a bug, automates or eliminates the root cause, and scales sublinearly.
Risky launch with a near-empty error budget?
Contain blast radius via canary and feature flags, use progressive rollout tied to budget burn, negotiate explicit risk acceptance, add fast rollback.
The Ironies of Automation: More Automation, More Problems?
Automating a system to reduce human error makes the human's role more critical, not less. The more reliable the automation, the less practice operators get for the rare, high-stakes moment it inevitably fails, leaving them unprepared to take control.
Customer Reliability Engineering: SRE for Your Customers
CRE extends SRE principles to your customers, making their reliability a shared goal. A provider partners with a key customer to co-engineer resilient services on their platform.
Just Culture: Fix the System, Not the Blame
Just Culture treats human error as a system failure, not a personal one. It's key for incident post-mortems, encouraging honest reporting to improve reliability.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles