tezvyn:

What is blast radius and how do you limit it?

AI-drafted, machine-checkedSource: interviewbeginner
WHAT IT TESTS

Safe production experimentation.

OUTLINE

Blast radius is the scope of users or systems an experiment can harm; limit it by targeting a small traffic percentage and by having an automated abort.

WHAT THIS TESTS Whether you understand that running experiments in production is acceptable only when the potential damage is deliberately contained and reversible.

A GOOD ANSWER COVERS Blast radius is the scope of impact a chaos experiment could have if it behaves worse than expected, measured in affected users, requests, dependencies, or regions. The core safety principle is to keep that radius small and grow it only as the experiment proves safe. Two concrete limiting techniques: first, scope the injection to a small slice, for example 1 to 5 percent of sessions, a single canary instance, or one region, leaving the majority of traffic untouched and providing a control group. Second, define guardrail metrics and an automated abort, a kill switch that immediately removes the fault and restores normal behavior the moment a key SLI breaches a hard threshold, rather than depending on a human noticing. Additional measures include running during business hours when staff are watching, having tested rollback, and starting in lower environments to gain confidence before production. The mindset is start small, monitor closely, and expand gradually.

COMMON WRONG ANSWERS Defining blast radius vaguely as just risk without the notion of scope. Limiting it only by running off-hours, when fewer staff are watching to react. Relying solely on a human to abort, which is slow. Jumping straight to full traffic to be realistic, defeating containment.

LIKELY FOLLOW-UPS How do you choose the initial percentage. What metrics trigger the abort. How fast must rollback be. How do you safely widen the radius over time.

ONE CONCRETE EXAMPLE You inject latency only into 2 percent of cart requests in one region during the workday. A monitor watches the cart success rate for that cohort, and a kill switch auto-removes the fault if success drops below 99 percent. Because only 2 percent of one region was ever exposed and the abort fired within seconds, the worst-case impact stayed tiny while you still learned how the system behaves under stress.

Read the original → principlesofchaos.org

Get five bites like this every day.

Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.