All bites
The whole library, newest first. Filter by what you are here for, or pick a topic if you already know.
8664 bites
Page 22
Key roles in incident response
Incident Commander coordinates and decides but does not fix; Communications Lead handles stakeholders; Operations/SMEs do hands-on diagnosis and remediation.
OpenTelemetry agent and gateway architecture
Agents run per-node for local collection and host enrichment; gateways are central, horizontally scaled pools for batching, tail sampling, and routing.
First-SRE 90-day plan at a startup
Listen and measure first, pick one high-impact service, define SLIs/SLOs and basic alerting, then show reduced toil or incidents to leadership.
Designing shallow vs deep health checks
Shallow checks confirm the process is alive; deep checks verify dependencies; use shallow for liveness/load-balancer routing and deep sparingly to avoid…
Rolling update vs blue-green deployment
Rolling replaces instances gradually with minimal extra capacity but mixes versions; blue-green runs two full environments for instant switch and rollback at double the cost.
Writing high-quality postmortem action items
Good action items are specific, assigned to an owner, prioritized, tracked to completion, and ideally prevent recurrence rather than just detect faster.
Calculating downtime for a 99.9% SLO
0.1% of 30 days is roughly 43 minutes of allowed downtime; healthy budget enables faster shipping while depletion slows or freezes deploys.
Designing an error budget policy
Define SLO and budget, tiered consequences as burn worsens, a feature freeze on exhaustion, and concrete earn-back criteria.
Keeping a postmortem blameless after an admission
Acknowledge the courage, redirect from who to why the system allowed it, ask what guardrails were missing.
Conducting a Production Readiness Review
Assess monitoring and alerting, capacity and load testing, failure modes and dependencies, on-call and runbooks, and rollback or release safety.
Embedded vs consulting SRE engagement models
Embedded SREs sit inside one team for deep impact but limited reach; consulting SREs advise many teams broadly but shallowly.
A team keeps blowing its error budget. First steps?
Analyze where the budget is burning via SLIs and postmortems, validate the SLO and SLIs are sound, then partner blamelessly on the top fixes.
What is an error budget and how is it used?
The budget is the allowed unreliability (100 percent minus the SLO); track its burn, ship freely when budget remains, and freeze risky changes to focus on reliability when exhausted.
Defining SLIs and an SLO for an auth service?
Pick user-centric SLIs like login availability and latency, measure good over valid events at the right boundary, then set an achievable SLO with a window.
Client-side chaos for an uncontrollable third party?
Inject faults at your client boundary via a proxy or fault-injecting wrapper, simulate timeouts, errors, and latency, then verify timeouts, retries, breakers, and fallbacks.
Chaos test for gray-failure cascades in shared services?
Inject partial latency into a shared service, hypothesize tenants stay isolated within steady state, and monitor cross-system queue depth, pool saturation, retries, and per-tenant SLIs.
Automating chaos in CI/CD for continuous verification?
Run codified experiments against staging or canary with pass/fail on steady-state SLIs; prerequisites are observability, automated abort, and isolation.
Resource faults versus network faults: when each matters?
Resource faults probe local saturation and autoscaling; network faults probe distributed-call resilience like timeouts and retries.
Why does 200ms latency drop requests? Diagnose it.
Little's Law shows added latency raises in-flight requests, exhausting the thread or connection pool; check pool saturation, timeouts, and retries.
How do you run your first production chaos experiment?
Pick a low-risk known weakness, define a measurable hypothesis, brief stakeholders and on-call, run small with an abort, then analyze and fix.