Load Balancing Algorithms: How to Pick a Server

Why it exists

Modern services are built for reliability and scale by running many identical, interchangeable server processes, often called backends. With hundreds or thousands of backends available, a system needs a logical way to distribute incoming work across them. Without a deliberate strategy, some servers would become overloaded while others sit idle, defeating the purpose of having a large pool.

The mental model

Think of a client as a dispatcher and the backends as a fleet of identical taxis. For every incoming ride request (a query), the dispatcher must decide which taxi to send. A naive policy is to just send the next taxi in line. A smarter policy is to send the taxi that is closest and currently has no passenger. Load balancing algorithms are these dispatching policies for network traffic.

How it works

When a client task needs something from a service, it consults a list of all available backend tasks for that service. It then runs an algorithm to select a single backend from that list and sends the query. This selection happens for each query, or for the life of a connection. The ideal goal is to spread the load perfectly evenly. In a complex system, this is a chained event: a frontend server picks an application server, which in turn acts as a client to pick a database server, with each step involving a load balancing decision.

When to use it

This is a fundamental pattern for any distributed system where multiple servers perform the same function. It's essential for stateless web applications, API gateways, microservice-to-microservice communication, and distributing reads across database replicas. If you have multiple homogeneous processes providing a service, you need a load balancing policy.

When not to use it

Load balancing is not needed for a service running on a single machine with no redundancy. It's also less applicable when backends are not interchangeable—for example, if each server is specialized for a different function. The core concept applies to pools of homogeneous, stateless servers.

One canonical example

Google's Frontend (GFE) is a reverse proxy system that handles all incoming HTTP requests. When a user's request arrives, the GFE acts as a client. It uses load balancing algorithms to select one specific backend process from the thousands running the target application. That backend processes the query. This same pattern repeats internally, as that backend might itself act as a client to call other services (like ads or authentication), using load balancing to pick backends for those services.