Design a simple chaos experiment for a cache dependency?

WHAT THIS TESTS Whether you can frame a falsifiable hypothesis about a cache dependency and pick the metrics that reveal whether the system degrades gracefully or cascades into failure.

A GOOD ANSWER COVERS State the steady state first, for example request success rate above 99.9 percent and p99 latency under 200ms. The hypothesis: if Redis becomes unavailable, the service falls back to reading from the origin database and remains within its steady state, accepting higher latency but not failures. Design the fault to be small and reversible, block connectivity to Redis or add latency for a limited slice of traffic or a single instance. Then monitor the right signals: request error rate and p99 latency to see user impact, the database's query load, CPU, and connection pool utilization to catch a thundering herd of cache-miss traffic stampeding the origin, and the cache hit and miss rates to confirm the fault took effect. Also watch for retry storms and timeouts. The key risk is that the cache silently absorbs most load, so losing it can overwhelm the database and cause a cascading failure rather than mere slowness.

COMMON WRONG ANSWERS Assuming the cache is purely optional, ignoring the load it offloads. Monitoring only the service's error rate and not the database's saturation. No hypothesis, so success is undefined. Failing the experiment by hitting all traffic at once and risking a real DB overload.

LIKELY FOLLOW-UPS What is a thundering herd and how do you mitigate it. Would you test cache latency versus full outage. How does request coalescing or a stale-while-revalidate strategy help. How do you bound the blast radius here.

ONE CONCRETE EXAMPLE You drop Redis connectivity for 5 percent of product-page requests. The hypothesis is that these still succeed via the database within latency limits. Dashboards show cache hit rate fall to zero for the cohort, p99 latency rise as expected, but database connections spike toward the pool limit, a warning that a full outage could exhaust it. The finding leads you to add request coalescing and a small in-process fallback cache before widening the test.