Retries and circuit breaking in a mesh

What's really being asked

Whether you can apply resilience patterns without code changes and, crucially, whether you understand how naive retries make things worse.

The full answer

In the mesh you declare retry behavior on the route, for example in an Istio VirtualService: set a small number of attempts, a per-try timeout, and a list of retriable conditions such as connect failures or 5xx, ideally only for idempotent requests. For circuit breaking you use the DestinationRule's connection pool settings and outlier detection. Connection-pool limits cap concurrent connections and pending requests so the caller fails fast rather than queueing indefinitely. Outlier detection watches each endpoint and, after a threshold of consecutive errors, ejects that pod from the load-balancing set for a cool-down, then probes it back. Together these stop a struggling dependency from dragging the caller down and give it room to recover.

The mistakes people make

Enabling unbounded retries with no per-try timeout, which multiplies load on an already failing service and triggers a retry storm. Retrying non-idempotent writes blindly, risking duplicates. Confusing a circuit breaker with a retry; the breaker stops sending traffic, the retry resends it. Forgetting jitter or backoff considerations and the interaction between retries at multiple hops, which compounds amplification.

What usually comes next

How do retries at several layers multiply? How do you choose idempotent versus non-idempotent handling? How does outlier detection differ from a Kubernetes readiness probe? What metrics tell you the breaker is opening, and how do you tune thresholds?

A concrete example

The orders service calls inventory. You set retries to two attempts with a 250 millisecond per-try timeout on connect-failure and 503 only. In the inventory DestinationRule you set outlier detection to eject any pod after five consecutive 5xx for thirty seconds and cap pending requests. When one inventory pod degrades, it is ejected and traffic shifts to healthy pods while the bounded retries cover brief blips, instead of every order request hammering the bad pod forever.