Designing shallow vs deep health checks

WHAT THIS TESTS: Whether you understand that a health check is not just 'is it up' but a control signal that decides whether traffic is routed to an instance, and that a poorly scoped check can take down a whole fleet.

A GOOD ANSWER COVERS: A shallow (or liveness) check answers whether the process itself is running and able to handle requests. It should be cheap and fast, return a simple OK without calling external dependencies, and is what a load balancer or orchestrator polls frequently to decide routing and restarts. A deep (or readiness/dependency) check verifies the things the service depends on, such as the database, cache, or critical downstream APIs. It is more expensive and more informative but dangerous on the routing path: if a shared dependency like a database has a transient blip, every instance fails its deep check at once and the load balancer removes the entire fleet, turning a minor dependency hiccup into a total outage. The design rule is to use shallow checks for load-balancer liveness and reserve deep checks for readiness gating at startup, periodic diagnostics, or dashboards, often with their own thresholds and dampening rather than instant ejection.

COMMON WRONG ANSWERS: Putting deep dependency checks directly on the load-balancer health path, returning 200 from a check that does nothing meaningful, or making the check so heavy it itself loads the dependency. Failing to distinguish liveness from readiness.

LIKELY FOLLOW-UPS: How do you prevent a single slow dependency from failing all instances? What is the difference between Kubernetes liveness and readiness probes here? How do you avoid the health check overloading the very dependency it checks?

ONE CONCRETE EXAMPLE: A service's load-balancer check originally pinged the primary database. When the database had a two-second hiccup, all instances failed simultaneously and the LB ejected the entire service, causing a full outage from a recoverable blip. The fix: the LB check becomes shallow (process alive), while dependency health is tracked separately with tolerance, so a brief database issue degrades gracefully instead of removing all capacity.