Latency Injection: Testing How Your System Handles Slowdown

Why it exists

In distributed systems, failures aren't always total crashes. Often, a service just becomes slow. Latency injection was created to simulate these "brownout" conditions, allowing engineers to test how their system handles delays before they happen in production and cause cascading failures.

The mental model

Think of latency injection as a controlled experiment in system slowness. You're a scientist introducing a specific delay into one part of a complex machine—like an API call or a database query—to observe the ripple effects. Does the calling service time out gracefully? Does it retry aggressively and make things worse? You're testing the system's patience.

How it works

Latency injection is a form of software-based fault injection. It's typically implemented using proxies, service mesh sidecars, or custom middleware in your application code. When a designated network request or function call occurs, the tool intercepts it and artificially adds a delay before passing it on to its destination or returning a response. This delay can be fixed (e.g., add 200ms) or variable.

When to use it

Use latency injection to test and validate specific resilience mechanisms. It's ideal for checking if your timeout settings are appropriate—not too short, not too long. It's also perfect for verifying that your circuit breakers trip correctly when a downstream service becomes sluggish, and that your retry logic doesn't create a "retry storm" that amplifies the problem.

When not to use it

Don't use latency injection as a substitute for load testing. While it tests how your system reacts to slowness, it doesn't tell you what causes the slowness in the first place. Also, avoid injecting extreme, unrealistic latencies in early stages of testing. A 30-second delay might prove a timeout works, but testing a more realistic 500ms delay might reveal more subtle bugs.

One canonical example

An e-commerce site wants to test what happens if its payment processing service becomes slow. Using a service mesh, engineers configure a rule to add a 500ms delay to all calls to the payment service. They run the test and observe that the "Place Order" button hangs with no feedback, leading to users clicking it multiple times. The test reveals they need a better loading indicator and to disable the button after the first click.