Design a chaos experiment for a payment dependency?
Rigorous chaos experiment design.
Define a measurable steady state, hypothesize it holds when payments fail, limit blast radius to a small traffic slice, and auto-abort on SLO breach.
What's really being asked
Whether you treat chaos as a disciplined experiment with a falsifiable hypothesis and strong safety controls, not as randomly breaking things in production.
The full answer
Define the steady state as a measurable user-facing metric, for example checkout completion rate above 99 percent and p99 latency under a threshold. State the hypothesis: even when the payment processor returns errors or high latency, checkout completion stays within the steady-state band because fallbacks, retries, and a circuit breaker absorb the fault. Pick the fault precisely, inject timeouts or 5xx into calls to the payment processor, possibly via a service mesh or fault-injection proxy. Limit the blast radius by scoping the injection to a small percentage of sessions or a single canary region, and run a control cohort untouched for comparison. Instrument guardrail metrics and wire an automated abort that removes the fault injection immediately if completion rate or error budget burn crosses a hard limit. After the run, compare cohorts, file findings, and fix gaps such as a missing fallback to a queued retry.
The mistakes people make
No measurable hypothesis, so you cannot tell success from failure. Injecting faults across all traffic at once, risking a real outage. No automated rollback, relying on a human to notice and react. Measuring only system metrics and ignoring the user-facing success rate. Skipping a control group, so you cannot attribute changes to the fault.
What usually comes next
How do you choose the blast-radius percentage. What triggers the auto-abort. How do you avoid charging real customers during the test. How do you build confidence to widen the radius.
A concrete example
In one region you inject 100 percent 503s into the payment processor for 2 percent of checkout sessions. The hypothesis is that these sessions still complete via a queued-authorization fallback at above 99 percent. A monitor watches completion rate per cohort; if it drops below 98 percent or error-budget burn spikes, the mesh rule is auto-removed within seconds. The run reveals the fallback queue lacks idempotency, a concrete fix you make before widening the experiment.
Interview question
What most clearly distinguishes a sound chaos experiment from reckless fault injection in production?
- a.It injects as many simultaneous faults as possible for realism
- b.It is performed without telling stakeholders to get honest results
- c.It is run only in staging, never production
- d.It has a measurable steady-state hypothesis, a bounded blast radius, and an automated abortCorrect
Why? this is the answer
Discipline comes from a falsifiable hypothesis, a limited blast radius, and an automatic stop on SLO breach. Maximizing simultaneous faults or hiding the test increases risk rather than rigor, and chaos can run safely in production with these controls.
Just read this? Test yourself on what you have been reading.
Read the original → principlesofchaos.org
- #chaos-engineering
- #resilience
- #blast-radius
- #steady-state
- #payments
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles