Game Days: Practice Breaking Your System Before It Breaks Itself

WHY IT EXISTS: Complex systems fail in unpredictable ways. Dashboards and plans are not enough; you need to know how the entire socio-technical system—people, processes, and code—reacts under real pressure. Game Days build this muscle memory safely.

THE MENTAL MODEL: A Game Day is a fire drill for your production environment. You don't wait for a real fire to discover your escape routes are blocked and alarms are broken. You simulate a disaster in a controlled way to find and fix problems when the stakes are low.

HOW IT WORKS: A planning team defines a failure scenario, like killing a database replica or taking down an availability zone. A response team (the on-call engineers) then uses existing tools and runbooks to detect, diagnose, and mitigate the issue while observers take notes. The goal isn't to "win," but to generate concrete action items for improving resilience. These exercises are often called Disaster Recovery Tests (DiRT).

WHEN TO USE IT: Use Game Days to validate runbooks, test on-call onboarding, verify monitoring for a critical service, or prepare for high-traffic events. They are especially valuable after a major architectural change or to test assumptions about system behavior.

WHEN NOT TO USE IT: Don't run a Game Day during a real incident or on a fragile system without extensive planning, stakeholder buy-in, and clear blast radius controls. Avoid them if the culture punishes failure; the exercise requires psychological safety to be effective. Start small, perhaps in staging, before targeting production.

ONE CANONICAL EXAMPLE: A classic DiRT exercise is simulating a regional failure. The planning team blocks all traffic to a cloud region, like us-east-1. The response team must then execute the failover procedure to us-west-2. The test reveals if DNS propagation is too slow, if data replication lags, if services have hardcoded regional dependencies, and if the on-call engineer even knows the failover runbook exists.