Reliability
101 bites tagged Reliability — interview questions with model answers, and 60-second explainers.
Essential sections of a post-mortem
Summary, impact, timeline, root cause, lessons, and owned action items, each with a clear purpose. Post-mortem structure. a narrative with no timeline or no concrete, tracked follow-ups.
Designing a safe chaos engineering exercise
Form a hypothesis, define steady state and blast radius, inject failure, observe response, with abort and rollback safeguards. Disciplined chaos experiments.
Fast rollback versus targeted fix in an incident
Weigh time-to-recovery and blast radius against rollback side effects; prefer restoring service fast, gated by data like deploy diff and migration risk. Mitigation decision-making under pressure.
Primary goal and sections of a post-mortem
Goal is organizational learning and prevention, not blame; include summary, timeline, impact, root cause, action items. Purpose of post-mortems. framing it as finding who to blame or a paperwork ritual.
Triaging a 5xx error spike
Confirm impact and scope, declare and assign roles, check recent changes, stop the bleeding before root cause. Calm, structured incident triage. diving into code fixes before assessing blast radius or mitigating.
Measuring on-call health quantitatively
Track pages per shift especially off-hours, actionability rate, time-to-resolve, and load distribution; use trends to prioritize reliability work and protect against burnout. turning on-call pain into data that justifies investment.
Setting SLOs across a dependency chain
Serial dependencies multiply, so each backend needs a tighter SLO than the target; mitigate with redundancy, caching, and graceful degradation. Reliability math across dependencies.
Handling planned maintenance in SLOs and error budgets
Either charge maintenance to the error budget or formally exclude it via a maintenance window; each trades honesty against fairness. Honest accounting of planned downtime.
Defining SLOs for a new critical service
Start from user journeys, pick SLIs, involve product, engineering, and business stakeholders, set realistic targets iteratively. Cross-functional SLO design.
What is an error budget?
Error budget is one minus the SLO, the allowed unreliability; it gates feature velocity versus reliability work. Linking reliability targets to delivery decisions.
Relationship between SLI, SLO, and SLA
SLI is the measured metric, SLO is the internal target for that metric, SLA is the external contract with consequences. Core reliability vocabulary. Treating the terms as synonyms or making the SLO looser than the SLA.
What makes a blameless postmortem effective?
Cover summary, impact, timeline, root cause, action items; explain blameless means focusing on systemic causes so people report honestly. understanding of postmortem structure and the psychology of blamelessness.
Error budget exhausted early: what now?
Invoke the error budget policy, shift focus from features to reliability, prioritize stability work, analyze what burned the budget. whether you use the error budget as a decision tool, not punishment.
How do you set SLOs for a service from scratch?
Identify critical user journeys, pick SLIs that reflect user happiness, measure current performance, set achievable targets with buffer, iterate. a disciplined, user-centric process for defining reliability targets.
SLI vs SLO vs SLA: how do they relate?
SLI is a measured metric, SLO is the internal target for that metric, SLA is the external contract with consequences. command of the core reliability vocabulary.
Normalization of Deviance
Normalization of deviance is the gradual process by which unsafe practices become accepted as normal because they have not yet caused a visible failure. Each tolerated shortcut lowers the bar, eroding safety margins until a catastrophe finally results.
Backpressure
Backpressure is a mechanism by which a slow consumer signals an upstream producer to slow down or stop, preventing unbounded queues and resource exhaustion. It keeps systems stable under overload by propagating capacity limits backward through a pipeline.
Review of Reviews (Postmortem Metrics)
A meta-process that audits the quality and follow-through of postmortems themselves, tracking metrics like action-item completion, time-to-close, and recurrence of incidents.
Actionable Remediation Items
Actionable remediation items are specific, owned, trackable follow-ups from a postmortem that prevent recurrence or improve response. Good ones are concrete and assigned with a due date; vague items like be more careful are non-actionable and predictably…
Incident Timeline
An incident timeline is a chronological, timestamped record of what happened during an incident: detection, key events, actions taken, and resolution. It anchors the postmortem in facts, separates symptoms from causes, and reveals detection and response…
Why design ML pipeline steps to be idempotent?
Re-running a step with the same input yields the same result and no duplicate side effects; enables safe retries and backfills. reliable, retryable pipeline design.
Handling a 401 error in an LLM agent's tool call
Catch the tool error, return a structured observation to the LLM, and distinguish recoverable retries from terminal failures needing re-plan or escalation. Robust agent error handling.
Prompt engineering to curb extraction hallucinations
Ground strictly in source, allow null for missing fields, enforce a schema, and use few-shot examples; acknowledge prompting cannot fully eliminate it. Practical hallucination control plus honesty about limits.
Fixing a prompt that ignores key constraints
Move the critical constraint to a prominent position, state it positively and specifically, separate instructions from data with delimiters, and add a concrete example. practical prompt-engineering instincts.
Get Reliability bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.