Advanced interview questions in Monitoring & SRE
Risky launch with a near-empty error budget?
Contain blast radius via canary and feature flags, use progressive rollout tied to budget burn, negotiate explicit risk acceptance, add fast rollback.
SRE vs traditional ops on a recurring alert?
Traditional ops repeatedly handles the alert manually and scales by adding people; SRE treats it as a bug, automates or eliminates the root cause, and scales sublinearly.
Error budget policy across dependent microservices?
Set per-service SLOs but anchor on user-facing journey SLOs, budget for dependency error via the multiplication of availabilities, attribute downstream-caused failures correctly, and use…
What is high-cardinality data in Prometheus?
Cardinality is the count of unique label combinations; each is a separate series; explosion blows up memory and query cost.
Head-based vs tail-based trace sampling
Head-based decides at trace start cheaply but blindly; tail-based waits for the full trace to keep errors and slow requests.
Symptom-based vs cause-based alerting
Symptom alerts fire on user-visible impact, cause alerts on internal conditions; page on symptoms to cut noise and catch unforeseen failures.
Setting SLOs across a dependency chain
Serial dependencies multiply, so each backend needs a tighter SLO than the target; mitigate with redundancy, caching, and graceful degradation.
Launching a risky feature with no error budget left
Bring burn history, what consumed the budget, blast radius, and rollback safety; offer mitigations like flags and canaries.
Fixing watermelon SLOs that hide unhappy users
Green-outside, red-inside SLOs come from server-side measurement, coarse aggregation, wrong events, and lax thresholds; fix by measuring at the client and using meaningful SLIs.
When to add custom OpenTelemetry instrumentation
Wrap domain logic in custom spans, attach business attributes like tenant or plan, answer revenue and per-customer questions.
Histograms versus summaries for latency
Histograms store bucket counts and stay aggregatable with query-time approximate quantiles; summaries precompute quantiles per instance that cannot be averaged.
Scaling Prometheus for HA and volume
Shard scraping, add long-term object storage and global query via Thanos/Cortex/VictoriaMetrics, run redundant replicas for HA, with dedupe.
Writing SLIs in PromQL
Availability is good requests over total using rate and non-5xx counters; latency uses histogram_quantile over rate of buckets summed by le.
Loki versus Elasticsearch for logs
Loki indexes only labels and stores raw log chunks, cheap but needs label-scoped brute-force search; Elasticsearch full-text indexes content, fast arbitrary search but costly to store…
Measuring on-call health quantitatively
Track pages per shift especially off-hours, actionability rate, time-to-resolve, and load distribution; use trends to prioritize reliability work and protect against burnout.
Catching rare intermittent failures
Add high-cardinality traces and structured logs with exemplars, use tail-based sampling to keep failing traces, and alert on SLO burn rather than coarse averages.
Alert silencing versus alert inhibition
Silencing mutes by matcher for a window, inhibition suppresses lower alerts when a higher one fires, dependency-aware.
Technical investments to reduce MTTR
Cut detection, diagnosis, and recovery time via observability, runbooks/automation, fast rollback, and resilient architecture.
Architecting a single source of truth for incidents
One curated view overlaying golden metrics, deploy/change events, and active alerts on a shared timeline, fed by standardized telemetry.
Designing a safe chaos engineering exercise
Form a hypothesis, define steady state and blast radius, inject failure, observe response, with abort and rollback safeguards.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles