FMEA: Systematically Mapping What Could Go Wrong

Why it exists

Complex systems fail in complex ways. Without a structured process, identifying potential weak points is often just guesswork. FMEA was created to replace reactive firefighting with a proactive, systematic method for analyzing and mitigating risks before they cause an outage.

The mental model

FMEA is like creating a detailed "risk encyclopedia" for your system. For every component—a database, an API gateway, a specific server—you create an entry that asks: "How can this break? Why would it break? And what happens to everything else when it does?" It forces you to think through the chain reaction of a failure before it happens.

How it works

The process involves reviewing system components, assemblies, and subsystems one by one. For each, the team identifies potential failure modes (e.g., "database returns slow queries"). Then, you analyze the potential effects ("downstream services time out") and causes ("missing index," "bad query plan"). This is all recorded in an FMEA worksheet. Teams often score each failure mode by severity, likelihood of occurrence, and detectability to create a Risk Priority Number (RPN), which helps prioritize mitigation efforts.

When to use it

Use FMEA during the design phase of a new system or before implementing a major change to an existing one. It's a powerful tool for building reliability in from the start, rather than trying to bolt it on later. It serves as the foundational first step for a comprehensive system reliability study.

When not to use it

FMEA is not a tool for incident response. During an active outage, you need a runbook, not a sprawling analytical document. It's also overkill for simple, non-critical systems where the failure modes are obvious and the impact is low, as the analysis can be time-consuming.

One canonical example

Component: User Authentication Service. Failure Mode: Latency spikes over 500ms. Potential Causes: Database connection pool exhaustion, slow upstream identity provider, or a new code change introducing an inefficient query. Effects: Users can't log in, leading to failed checkouts. Dependent services requiring auth will also fail, cascading the issue. The FMEA process would flag this risk, and the team might prioritize adding better connection pool monitoring and circuit breakers as a result.