Design a chaos experiment for a payment dependency?

WHAT THIS TESTS Whether you treat chaos as a disciplined experiment with a falsifiable hypothesis and strong safety controls, not as randomly breaking things in production.

A GOOD ANSWER COVERS Define the steady state as a measurable user-facing metric, for example checkout completion rate above 99 percent and p99 latency under a threshold. State the hypothesis: even when the payment processor returns errors or high latency, checkout completion stays within the steady-state band because fallbacks, retries, and a circuit breaker absorb the fault. Pick the fault precisely, inject timeouts or 5xx into calls to the payment processor, possibly via a service mesh or fault-injection proxy. Limit the blast radius by scoping the injection to a small percentage of sessions or a single canary region, and run a control cohort untouched for comparison. Instrument guardrail metrics and wire an automated abort that removes the fault injection immediately if completion rate or error budget burn crosses a hard limit. After the run, compare cohorts, file findings, and fix gaps such as a missing fallback to a queued retry.

COMMON WRONG ANSWERS No measurable hypothesis, so you cannot tell success from failure. Injecting faults across all traffic at once, risking a real outage. No automated rollback, relying on a human to notice and react. Measuring only system metrics and ignoring the user-facing success rate. Skipping a control group, so you cannot attribute changes to the fault.

LIKELY FOLLOW-UPS How do you choose the blast-radius percentage. What triggers the auto-abort. How do you avoid charging real customers during the test. How do you build confidence to widen the radius.

ONE CONCRETE EXAMPLE In one region you inject 100 percent 503s into the payment processor for 2 percent of checkout sessions. The hypothesis is that these sessions still complete via a queued-authorization fallback at above 99 percent. A monitor watches completion rate per cohort; if it drops below 98 percent or error-budget burn spikes, the mesh rule is auto-removed within seconds. The run reveals the fallback queue lacks idempotency, a concrete fix you make before widening the experiment.