All-Reduce: Synchronizing Parallel Workers
All-Reduce lets parallel workers agree on a global result. Each worker contributes data, an operation (like sum) runs on all data, and every worker gets the final answer. It's the core of distributed ML training, used to average gradients across GPUs.
WHY IT EXISTS In parallel computing, workers often operate on just one slice of a problem. To make global decisions or synchronize state, they need a way to combine their local results and ensure every worker has the final, aggregated value. Without this, they can't proceed in lockstep.
THE MENTAL MODEL Think of a board meeting where a budget needs approval. Each department head (a worker) brings their own proposed spending (local data). An operation, 'sum,' is performed on all proposals. The final, total budget (the reduced result) is then written on the main whiteboard for every department head to see and use (the 'all' part). Every participant both contributes and receives the final result.
HOW IT WORKS A naive approach sends all data to one root node for computation and broadcast, creating a bottleneck. A common, more efficient method is Ring All-Reduce. Workers are arranged in a logical ring and data is chunked. In a 'scatter-reduce' phase, chunks are passed around the ring, with each node adding its data. This continues until each node holds a fully reduced chunk. In a second 'all-gather' phase, these final chunks are circulated so every node can reassemble the complete final result, balancing network load.
WHEN TO USE IT This is the workhorse of distributed data-parallel training in machine learning. Frameworks like Horovod and PyTorch's DistributedDataParallel use it to average gradients calculated on different GPUs. It is also fundamental in High-Performance Computing (HPC) for large-scale simulations requiring global state synchronization.
WHEN NOT TO USE IT If only one process needs the final result, a simpler 'Reduce' operation is more efficient, as it omits the final broadcast-to-all step. If no aggregation is needed and you just need to send data from one node to all others, use a 'Broadcast'. All-Reduce is specifically for many-to-many aggregation and distribution.
ONE CANONICAL EXAMPLE Training a model on 8 GPUs. Each GPU processes a different data batch and computes a gradient vector. An All-Reduce operation sums the 8 gradient vectors. The resulting sum is delivered back to all 8 GPUs. Each GPU then divides this sum by 8 to get the average gradient and updates its local copy of the model weights identically, ensuring the model stays synchronized across all workers.
Read the original → en.wikipedia.org
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.