Chaos test for gray-failure cascades in shared services?

What's really being asked

Whether you understand that gray failures, partial degradation rather than clean outages, are the hardest to detect and the most likely to cascade, and that shared multi-tenant services concentrate that risk.

The full answer

Design the fault as a partial degradation, not a kill: inject elevated latency, say a percentile-targeted slowdown, or a small fraction of errors into the shared service, optionally affecting only some requests, mirroring a real gray failure that health checks may still report as healthy. The steady-state hypothesis spans systems: tenant isolation holds so one tenant's load or the degradation does not breach other tenants' SLIs, and downstream and upstream dependents stay within their steady state because timeouts, bounded retries, bulkheads, and circuit breakers contain the slowdown. Monitor across multiple systems, not just the injected one: per-tenant latency and error SLIs to detect noisy-neighbor or cross-tenant blast, caller-side thread and connection pool saturation and queue depths to catch Little's Law amplification, retry rates and circuit-breaker open or half-open state to see whether the system is absorbing or amplifying, and resource saturation on shared infrastructure. Distributed traces tie the chain together. Crucially, watch tail percentiles, not averages, since gray failures hide in the tail.

The mistakes people make

Injecting a hard down instead of partial degradation, missing the gray-failure class. Monitoring only the injected service's average metrics, which mask both tail latency and cross-tenant impact. Ignoring retry amplification and pool saturation in callers. Assuming tenant isolation without measuring per-tenant SLIs.

What usually comes next

Why do averages hide gray failures. How does retry amplification cause cascades. How do you ensure tenant isolation, quotas, bulkheads, fair queuing. What makes gray failures hard to detect with health checks.

A concrete example

In a shared metadata service used by many tenants, you inject 300ms of added latency on 10 percent of requests for one tenant's traffic. The hypothesis is that other tenants' p99 stays flat and no caller's pool saturates. Dashboards reveal that a caller's bounded thread pool fills as Little's Law predicts, its retries amplify load on the shared service, and a second tenant's p99 climbs, exposing a missing bulkhead and a retry storm, a cascading gray failure caught before it hit production.