LitmusChaos

WHY IT EXISTS Distributed systems fail in ways integration tests never exercise: a node dies, latency spikes, a dependency times out. Teams need to verify resilience proactively rather than discovering weaknesses during a real outage. LitmusChaos exists to make injecting those failures safe, repeatable, and native to Kubernetes.

THE MENTAL MODEL Treat chaos as a controlled experiment with a hypothesis. You assert a steady state, such as the service stays available, then deliberately break something and check whether the assertion still holds. LitmusChaos encodes the hypothesis, the fault, and the verdict as Kubernetes objects so the experiment is version-controlled and reproducible.

HOW IT WORKS A ChaosExperiment defines a reusable fault pulled from ChaosHub. A ChaosEngine binds that experiment to a target workload and configures parameters like duration and blast radius. Probes run before, during, and after to validate steady state, for example an HTTP probe checking availability. The chaos operator orchestrates injection, collects results, computes a resilience score, and records a pass or fail verdict.

WHEN IT MATTERS It matters for stateful or high-availability services on Kubernetes, before launches, and as a continuous gate in CI/CD so regressions in resilience are caught early. Limiting blast radius is essential to run it safely in or near production.

ONE CONCRETE EXAMPLE A team believes their three-replica service tolerates losing one pod. They define a pod-delete ChaosEngine targeting one replica, with an HTTP probe asserting the endpoint returns 200 throughout. The experiment runs in staging and the probe fails: a misconfigured readiness check routed traffic to the terminating pod for several seconds. They fix the probe and graceful shutdown, rerun, and now achieve a full resilience score, having validated the assumption before it mattered in production.