Designing a safe chaos engineering exercise

WHAT THIS TESTS Whether you treat chaos engineering as a scientific, safeguarded practice that validates response, not as breaking things for thrills.

A GOOD ANSWER COVERS Frame it as an experiment. Define steady state with a measurable metric, then a hypothesis such as if a database replica fails, the system fails over within 30 seconds and on-call is paged. Start with the smallest meaningful blast radius, a single instance, one availability zone, or a small percent of traffic, and expand only as confidence grows. Inject a realistic fault and observe end to end: did monitoring detect it, did the right alert fire, did the runbook work, did responders coordinate. The point is testing the response process and tooling, not just the system. Safeguards are essential: a fast kill switch, automated abort if SLOs degrade past a threshold, running off-peak, notifying stakeholders, ensuring backups and rollback are ready, and never experimenting blind.

COMMON WRONG ANSWERS Randomly terminating production with no hypothesis or metric. No blast-radius limit, so an experiment becomes a real outage. No abort mechanism.

LIKELY FOLLOW-UPS How do you run game days versus automated continuous chaos? How do you build org trust to experiment in prod? What do you do with findings?

ONE CONCRETE EXAMPLE You hypothesize that losing one cache node degrades gracefully. Off-peak, with stakeholders notified and an abort rule set to halt if error rate exceeds 1 percent, you kill a single cache node in one zone. You watch whether the alert fires, whether the runbook is found and accurate, and whether the on-call responds in time. You discover the alert never fired and the runbook was stale. The system stayed healthy, so no real impact, and you file fixes for both gaps before widening the experiment.