What is an error budget and how is it used?
WHAT IT TESTS: Understanding error budgets as a release governor. OUTLINE: The budget is the allowed unreliability (100 percent minus the SLO); track its burn, ship freely when budget remains, and freeze risky changes to focus on reliability when exhausted.
Defining SLIs and an SLO for an auth service?
WHAT IT TESTS: Translating user needs into measured reliability. OUTLINE: Pick user-centric SLIs like login availability and latency, measure good over valid events at the right boundary, then set an achievable SLO with a window.
Client-side chaos for an uncontrollable third party?
WHAT IT TESTS: Testing resilience you cannot reach directly. OUTLINE: Inject faults at your client boundary via a proxy or fault-injecting wrapper, simulate timeouts, errors, and latency, then verify timeouts, retries, breakers, and fallbacks.
Chaos test for gray-failure cascades in shared services?
WHAT IT TESTS: Detecting subtle cascading failure. OUTLINE: Inject partial latency into a shared service, hypothesize tenants stay isolated within steady state, and monitor cross-system queue depth, pool saturation, retries, and per-tenant SLIs.
Automating chaos in CI/CD for continuous verification?
WHAT IT TESTS: Operationalizing chaos as a gate. OUTLINE: Run codified experiments against staging or canary with pass/fail on steady-state SLIs; prerequisites are observability, automated abort, and isolation.
Resource faults versus network faults: when each matters?
WHAT IT TESTS: Matching fault type to a system's real risks. OUTLINE: Resource faults probe local saturation and autoscaling; network faults probe distributed-call resilience like timeouts and retries.
Why does 200ms latency drop requests? Diagnose it.
WHAT IT TESTS: Reasoning about concurrency limits and queueing. OUTLINE: Little's Law shows added latency raises in-flight requests, exhausting the thread or connection pool; check pool saturation, timeouts, and retries.
How do you run your first production chaos experiment?
WHAT IT TESTS: End-to-end experiment process and judgment. OUTLINE: Pick a low-risk known weakness, define a measurable hypothesis, brief stakeholders and on-call, run small with an abort, then analyze and fix.
Design a simple chaos experiment for a cache dependency?
WHAT IT TESTS: Applying chaos thinking to a cache. OUTLINE: Hypothesis that the service degrades gracefully when Redis is unavailable, monitor error rate, latency, DB load, and cache hit rate.
What is blast radius and how do you limit it?
WHAT IT TESTS: Safe production experimentation. OUTLINE: Blast radius is the scope of users or systems an experiment can harm; limit it by targeting a small traffic percentage and by having an automated abort.
How does chaos engineering differ from other testing?
WHAT IT TESTS: Conceptual grasp of chaos engineering. OUTLINE: It experiments on real systems by injecting faults to test a steady-state hypothesis, versus verifying known behaviors like integration or load tests.
Reliability paved roads for an internal PaaS?
WHAT IT TESTS: Platform thinking that makes reliability the default. OUTLINE: Built-in observability, safe deploys with health checks and rollback, sane timeouts/retries/limits, and SLO tooling.
Design a chaos experiment for a payment dependency?
WHAT IT TESTS: Rigorous chaos experiment design. OUTLINE: Define a measurable steady state, hypothesize it holds when payments fail, limit blast radius to a small traffic slice, and auto-abort on SLO breach.
Replication and consistency for active-active regions?
WHAT IT TESTS: Distributed consistency trade-offs. OUTLINE: Choose per data class between synchronous (low RPO, higher latency) and async replication, address write conflicts, and reason via CAP and PACELC.
Zero-downtime index migration on a hot table?
WHAT IT TESTS: Safe schema change at scale. OUTLINE: Build the index concurrently to avoid table locks, run off-peak with monitoring, and keep it reversible since dropping an index is cheap. RED FLAG: A blocking CREATE INDEX that locks writes on a hot table.
Mitigating risk from an unproven external dependency?
WHAT IT TESTS: Designing for dependency failure. OUTLINE: Timeouts and circuit breakers to fail fast, bulkheads to isolate resources, fallbacks or cached/degraded responses.
Reliability patterns for queue-based job processing?
WHAT IT TESTS: Designing fault-tolerant async workers. OUTLINE: Retries with backoff and jitter for transient faults, dead-letter queues plus a poison-message limit, idempotent handlers and visibility timeouts.
What is an SLO and how do you define API availability?
WHAT IT TESTS: Understanding SLI versus SLO and how to measure reliability. OUTLINE: SLO is a target on an SLI, availability SLI is good requests over valid requests, you need labeled request counts over a window.
What telemetry must a new microservice ship with?
WHAT IT TESTS: Practical observability literacy. OUTLINE: Metrics for trends like p99 latency and error rate, structured logs for per-request detail, traces for cross-service causality. RED FLAG: Listing only host CPU dashboards and no request-level signals.
When horizontal scaling is the wrong fix
WHAT IT TESTS: Scaling judgment. OUTLINE: adding instances fails when the bottleneck is a shared resource like one database, a lock, or a queue, so more instances just add contention; investigate where time is actually spent.