Reliability
101 bites tagged Reliability — interview questions with model answers, and 60-second explainers.
The ACID properties of transactions
Define Atomicity, Consistency, Isolation, Durability and why each matters. foundational transaction guarantees. confusing Consistency with Isolation or thinking durability means in-memory only.
Idempotency in data ingestion pipelines
Idempotency means re-running a step yields the same result with no duplicates; it matters because retries and at-least-once delivery are inevitable; achieve it with deduplication keys or upserts. reliability under retries.
Monitoring with SLOs and error budgets
Define SLIs from the user's view, set SLO targets, derive an error budget, and alert on burn rate rather than raw thresholds. SRE reliability targets. paging on every CPU blip with no link to user impact.
Idempotency in event-driven systems
Idempotency means repeated processing yields the same end state; it matters because messages get redelivered; achieve it with idempotency keys and conditional writes. handling at-least-once delivery. assuming exactly-once.
Purpose and setup of a Dead-Letter Queue
A DLQ captures messages that repeatedly fail so they neither block the queue nor get lost; configure a redrive policy with a max receive count and alarm on it. handling unprocessable messages. silently dropping failures.
Mesh control plane vs data plane availability
The control plane configures and distributes policy; the data plane is the sidecars carrying traffic with cached config, so a brief control-plane outage keeps existing traffic flowing but… mesh plane separation and failure modes.
Impact of losing etcd quorum
Without quorum etcd goes read-only-ish and the API server cannot persist writes, so scheduling and changes stall, but kubelets keep running existing pods. control vs data plane separation.
How do you reliably deliver a survey to 10,000 users?
Tests systems thinking for outreach at scale. Strong answers cover idempotent delivery, rate-limited async sending, PII handling and retention, plus observability into delivery and response rates.
Design a measurement framework and experimentation plan for a risky feature rollout
This tests balancing upside against operational risk. A strong answer defines guardrail metrics for stability and cost, sequences canary before A/B tests, and sets rollback thresholds. A red flag is ignoring latency or cost to chase engagement lift.
Operational vs Programmer Errors in Node
Operational errors are expected problems like a failed network request; programmer errors are bugs like reading undefined. Handle the first gracefully, crash the second. The footgun is catching programmer errors and continuing, which corrupts process state.
SLOs Tied to User Journeys, Not APIs
A user-journey SLO measures the full flow a person experiences, not one microservice's health. If checkout is 99.9% up but payments fail, the metric lied. Teams drown in green per-service dashboards while users are furious.
DevOps Is Culture, SRE Is Engineering
DevOps is a cultural philosophy for fast, safe delivery; SRE is the engineering discipline that implements it with error budgets and SLOs. They are complementary, not rival job titles. The footgun is hiring SREs and declaring DevOps done.
What fixes an LLM agent's incorrect JSON arguments for a complex tool?
Tests mixing prompting with system guardrails for valid tool JSON. Outline: few-shot demos plus CoT prompting; schema validation, constrained decoding, and retries. Red flag: weak prompts without validation or structured output.
Design a rate-limited REST API data collection script
Tests client-side throttling discipline versus reactive 429 handling. Strong answers proactively pace calls using rate-limit headers, cap concurrency, and apply exponential backoff with jitter. Red flag: tight-loop retries or ignoring headers.
Build a system to send 1M personalized emails in 2 hours
This tests async distributed throughput and deliverability. Cover partitioned queues, auto-scaling workers, per-ESP rate limits, IP rotation, and exponential backoff with dead-letter queues. Red flags: synchronous sends, skipping IP warmup, or no retry logic.
Rolling vs blue/green deployments: differences and trade-offs
Tests risk-cost-downtime judgment. Contrast rolling's gradual swap with blue/green's parallel swap; rolling is cheaper but mixes versions, while blue/green doubles capacity for instant rollback. Red flag: saying blue/green cheaper or rolling double-capacity
APM: Turning System Metrics into Business Meaning
APM turns system metrics into business meaning by tracking software performance and availability. It matters most when slowdowns threaten service levels. The footgun is gathering data without translating IT metrics into what the business actually cares about.
Guarantee at-least-once delivery for a critical analytics event?
Tests reliable messaging patterns to avoid dual-write issues. Propose the Transactional Outbox pattern: atomically write business data and the event to a DB outbox table. A separate relay process then sends the event.
Define idempotency in data processing and give an example
Tests your grasp of distributed systems reliability. Define idempotency (N>1 runs = 1 run), explain its role in fault-tolerant retries, and provide a concrete example using transaction IDs. A red flag is confusing it with immutability.
Escalation Policy: When to Stop Shipping and Start Fixing
An escalation policy is a pre-agreed plan for when to divert engineers from feature work to fix reliability. When a service's error budget burns too fast, the policy's thresholds trigger specific actions. The footgun is thinking a quick rollback is enough.
Customer Reliability Engineering: SRE for Your Customers
CRE extends SRE principles to your customers, making their reliability a shared goal. A provider partners with a key customer to co-engineer resilient services on their platform.
SRE Office Hours: Scaling Expertise Through Consultation
SRE Office Hours are a scheduled time for development teams to get expert advice from SREs, scaling reliability knowledge without embedding them everywhere. Use them to review designs or define SLOs. The footgun is treating them as on-call support.
The SRE Mandate: Breaking the Dev vs. Ops Cycle
SRE breaks the classic conflict between developers who want to ship fast and operations teams who want stability. It treats operations as a software problem, not a human one. The footgun is creating a rebranded sysadmin team that still does manual work.
FMEA: Systematically Mapping What Could Go Wrong
FMEA is a structured pre-mortem for system components, cataloging potential failures, their causes, and their effects. It's used in system design to proactively identify risks before they become outages. The footgun is treating it as a one-time task.
Get Reliability bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.