Load Balancer Health Checks: Don't Route to Dead Servers

WHY IT EXISTS To automate failure detection in a pool of servers. Without health checks, a load balancer would blindly distribute traffic using its routing algorithm, sending a percentage of users to servers that have crashed or are frozen. This would result in user-facing errors and service degradation. Health checks make a load balancer an active manager of backend health, not just a passive distributor of traffic.

THE MENTAL MODEL Think of a restaurant host managing a team of waiters. The host periodically glances at each waiter's section. If a waiter is overwhelmed or has gone on an unexpected break (unhealthy), the host stops seating new tables in their section. Once the waiter catches up or returns (healthy), the host resumes seating guests there. The load balancer is the host, the servers are the waiters, and the health check is that periodic glance.

HOW IT WORKS A load balancer is configured to send a request to each backend instance on a specific port and path (e.g., an HTTP GET to /healthz). It expects a specific response, typically an HTTP 200 OK status code, within a set timeout. If an instance fails a certain number of consecutive checks (the "unhealthy threshold"), the load balancer marks it as unhealthy and stops sending it new connections. When the instance starts passing checks again for a certain number of times (the "healthy threshold"), it's put back into the active rotation.

WHEN TO USE IT Always use health checks when you have more than one backend instance behind a load balancer. They are critical for achieving high availability and fault tolerance in web applications, microservices architectures, and any distributed system. They are the foundation of self-healing infrastructure, allowing a system to automatically recover from single-server failures without manual intervention.

WHEN NOT TO USE IT It is almost never correct to disable health checks in a production multi-server environment. The only exception might be a temporary, single-instance setup for development or debugging where you explicitly want traffic to hit the instance regardless of its state. Disabling them in production negates a primary benefit of using a load balancer.

ONE CANONICAL EXAMPLE An Application Load Balancer manages traffic for three web servers. Its health check is configured to send an HTTP GET to /status on each server every 10 seconds. The unhealthy threshold is 3 failures, and the healthy threshold is 2 successes. If Server #2's application crashes, it stops responding to /status. After 30 seconds (3 failed checks), the load balancer stops sending any new user traffic to Server #2. When an engineer reboots the application, it starts passing checks again. After 20 seconds (2 successful checks), the load balancer automatically adds it back to the active pool.