Reliability
101 bites tagged Reliability — interview questions with model answers, and 60-second explainers.
Threat Modeling for Reliability: Find Failures Before They Happen
Threat modeling for reliability is like pre-gaming an outage: you systematically ask "how will this break?" before shipping code. Instead of attackers, you model failures like a database failing or a cloud region going down, then design mitigations.
N+1 Redundancy: One Spare for the Whole System
N+1 redundancy is like having one spare tire for your whole car. You have N active components and one shared, passive backup. It's used in server racks or power supplies for high availability without doubling costs.
Single Point of Failure: Your System's Achilles' Heel
A single point of failure (SPOF) is your system's Achilles' heel—one component whose failure causes a total outage. This is critical in high-availability design, like ensuring a website survives a server crash. The footgun is missing implicit SPOFs.
Soak Testing: Finding Bugs That Only Time Reveals
Soak testing is like running a marathon, not a sprint, to find bugs that only time reveals. It applies a typical production load over a long period to uncover slow memory leaks or resource exhaustion. The footgun is confusing it with stress testing.
Stress Testing: Finding Your System's Breaking Point
Stress testing finds a system's breaking point by pushing it beyond normal limits. It's used for mission-critical software to test robustness and availability under extreme load, not just normal function.
Load Testing: Simulating Real-World User Traffic
Load testing answers 'Can our system handle expected traffic?' by simulating many users at once. Use it before a big launch to find bottlenecks. The footgun is confusing it with stress testing, which pushes a system past its limits to see how it breaks.
Auto-Remediation: Automated Fixes for Common Failures
Auto-remediation is a system's immune response, automatically detecting and fixing known problems like a crashed service. It's a core SRE practice for improving availability, but a bad script can create a 'remediation storm' that worsens an outage.
On-Call Rotations: Engineering Reliability Under Pressure
On-call rotations are the human backstop for service reliability, with engineers responding to alerts in minutes. This is critical for high-availability services like search or email.
SLO Document: Your Service's Reliability Contract
An SLO document is the source of truth for a service's reliability promises, acting as a formal contract between teams on what 'good enough' looks like. It defines measurable targets for availability and latency, guiding engineering priorities.
SLO Time Windows: Choosing Your Measurement Period
An SLO's time window is the calendar for grading reliability. A short window forces rapid fixes, while a long one smooths out blips for strategic planning. The footgun is using calendar months, which have unequal lengths and complicate budget math.
Error Budgets: The Currency of Reliability
An error budget is the acceptable amount of downtime or errors your service can have over a period. Calculated as (1 - SLO), it's a currency for balancing risk (new features) and reliability work. The main footgun is treating it as a target to spend.
SRE Engagement Models: From Gatekeeper to Platform Builder
SRE engagement models define how reliability experts help product teams, evolving from gatekeeping existing services to providing reliable platforms. This applies when scaling an SRE team's impact.
Error Budgets: Balancing Reliability and Innovation
An error budget is your service's allowance for unreliability, calculated as 1 minus the SLO. It provides a data-driven signal for when to halt new feature releases and focus on stability, protecting users from repeated SLO misses.
Embracing Risk: Reliability Is a Budget, Not a Goal
Instead of chasing 100% uptime, SRE treats reliability as a budget. The acceptable downtime, or 'error budget,' is permission to take calculated risks like shipping features faster. The footgun is wasting resources on reliability users won't even notice.
Site Reliability Engineering (SRE): Ops as a Software Problem
Site Reliability Engineering (SRE) treats operations as a software problem, using engineering to automate and scale system management. It's crucial for massive services like Google Search, ensuring availability, latency, and capacity.
Kubernetes Probes: Liveness, Readiness, and Startup
Kubernetes probes ask your app about its health. Liveness asks 'are you alive?' (restart if not), readiness asks 'can you take work?' (pause traffic if not), and startup protects slow-starting apps. This is key for self-healing and zero-downtime deployments.
Your Design System's SLA: The Contract for Reliability
A Design System SLA is a contract defining the system's reliability and support promises. It specifies uptime guarantees, support hours, and incident response times, helping you assess if the system is a dependable foundation for your product.
Database High Availability: Surviving Server Failure
High Availability (HA) means having a hot standby database ready to take over instantly upon failure. It's essential for critical systems like payment gateways where downtime is unacceptable.
Database Transaction Log: Your System's Safety Net
A transaction log is your database's safety journal. Before changing data, it records the intended action in a durable file. This is vital for crash recovery, ensuring data isn't left corrupt. The footgun: its primary role is integrity, not just auditing.
Idempotency: Making Data Pipelines Retry-Safe
Idempotency means an operation has the same effect whether run once or multiple times, like closing an already-closed door. It's essential for data pipelines where retries are common. The footgun is assuming retries are safe, leading to data corruption.
Cloud Disaster Recovery: Planning for Failure
Cloud DR isn't about backups; it's a business continuity plan defining acceptable downtime (RTO) and data loss (RPO). It's for when a whole region fails or a bad deploy corrupts data. The footgun is having a plan but never testing it, creating false security.
SLIs & SLOs: Measuring What Matters for Service Reliability
SLIs are what you measure (e.g., latency); SLOs are the target you aim for (e.g., 99% success). They replace vague feelings about service health with concrete numbers. This is how SREs define and manage reliability.
Dead-Letter Queues: A Message Queue's Lost-and-Found
A Dead-Letter Queue (DLQ) is a message queue's lost-and-found, catching messages that can't be delivered. It's used to isolate "poison pill" messages that crash consumers or messages that have expired.
Data Durability vs. Availability: Lost vs. Unreachable
Availability is "can I reach my data right now?" while durability is "will my data still exist tomorrow?". Cloud storage systems are designed for both, but they solve different problems: network outages vs. data corruption.
Get Reliability bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.