Client-side chaos for an uncontrollable third party?

What's really being asked

Whether you realize resilience to a black-box dependency is tested at your own boundary, not theirs, and can design that client-side experiment concretely.

The full answer

The key reframe: you cannot break the third party, but you can simulate their failures at the point where your code calls them. Insert fault injection at the client boundary, a wrapper around the client library, a sidecar or service-mesh rule, or an interception proxy that sits between you and the provider. Reproduce the provider's realistic failure modes: connection errors and 5xx, request timeouts and elevated latency, rate-limit and throttling responses, partial or malformed payloads, and intermittent failures. The steady-state hypothesis asserts graceful degradation, that when the provider misbehaves your service still meets its SLIs via timeouts, bounded retries with backoff and jitter, a circuit breaker that fails fast, idempotent operations, and a fallback such as a cached value, queued retry, or degraded experience. Monitor your own error rate, latency, thread and connection pool saturation, retry counts, and breaker state, and confirm retries do not stampede the provider or breach its rate limits. Use a small blast radius, a traffic slice or canary, and an automated abort. For safety, target the provider's sandbox or test endpoints where possible.

The mistakes people make

Concluding the dependency is untestable because you do not own it. Only testing a hard down and ignoring latency, throttling, and malformed responses. Forgetting that aggressive retries can violate the provider's rate limits. Lacking any fallback, so the breaker just converts slowness to errors with no graceful path.

What usually comes next

How do you simulate throttling versus timeout. Where exactly do you inject, library, sidecar, or proxy. How do you avoid hitting the provider's real rate limits during the test. What fallback fits a payment gateway.

A concrete example

For a payment gateway you put a fault-injecting proxy in front of the client and, for 2 percent of checkout sessions, return 503s and 5 second latencies. The hypothesis is that these sessions degrade to a queued-authorization fallback while staying above the checkout SLO. Monitoring shows your connection pool nearing saturation and retries climbing, revealing that the timeout is too long and retries are unbounded, concrete fixes that harden you against a real gateway outage you could never trigger directly.