Inference Health Checks: Traffic Gates, Not Heartbeats

Why it exists

Machine learning inference servers wrap large model artifacts, GPU drivers, and complex initialization sequences. If a load balancer sends traffic to an instance that has finished booting but has not yet loaded a model into VRAM, requests will fail or hang. Health checks solve this coordination problem by giving the orchestrator a binary signal about whether an instance is safe to use before it receives real traffic.

The mental model

Think of the inference server as a restaurant kitchen. An unlocked front door does not mean the kitchen is ready to take orders; the chefs might still be unpacking ingredients. A proper health check is the head chef telling the host stand that service can begin. In distributed systems, the load balancer is the host stand, and the health check is the only message it trusts.

How it works

A typical setup exposes two endpoints. The liveness probe asks if the process is still running, and if it fails the orchestrator kills the container. The readiness probe asks if the process is willing and able to accept traffic right now. For inference, readiness should confirm that the model weights are loaded, the GPU context is initialized, and optionally that a warm-up inference has completed within a latency threshold. Some teams implement a deep health check that runs a dummy prediction, but this consumes compute and can skew production metrics. The probe is usually an HTTP GET polled every few seconds by Kubernetes or a cloud load balancer.

When to use it

Use a readiness probe whenever model loading is slower than container startup, which is almost always in GPU-based serving. Use it when running multiple replicas behind a load balancer so that rolling updates do not shift traffic to a replica that is still initializing. Also use it when your server has a dynamic batching queue that can become saturated; a custom readiness check can temporarily remove the instance from the pool until the queue drains.

When not to use it

Do not use an expensive deep inference probe on every node in a massive fleet if the check itself consumes a significant fraction of your GPU budget. Do not rely solely on the default root path of a serving framework unless you have verified that it fails when the model repository is empty. Do not set the probe interval so aggressively that the health endpoint starves the inference threads.

One canonical example

A computer vision team deploys a ResNet model via NVIDIA Triton on a Kubernetes cluster. Their readiness probe hits Triton's v2/health/ready endpoint, which returns HTTP 200 only after all models have loaded and GPU memory allocation has succeeded. During a rolling deployment, Kubernetes keeps the old pods active until the new pods report ready, preventing requests from reaching an instance that is still pulling a multi-gigabyte model from S3. When a node experiences a CUDA out-of-memory error, the readiness probe fails and the load balancer stops sending traffic while the liveness probe keeps the pod alive just long enough for an operator to inspect the logs.