Pipeline Parallelism: An Assembly Line for Your Model

Why it exists

Modern AI models can have billions or trillions of parameters, far exceeding the memory of a single GPU. While data parallelism replicates the model on each GPU and splits the data, pipeline parallelism splits the model itself, allowing it to span multiple devices.

The mental model

Think of it as a factory assembly line for your model's layers. Each GPU is a worker at a station responsible for a specific set of tasks, which is a 'stage' of layers. A data batch is broken into small parts, or micro-batches, that move down the line. As one worker finishes its task on a part and passes it on, it immediately starts on the next part, keeping the whole line busy.

How it works

The model's layers are partitioned into sequential stages, and each stage is assigned to a different GPU. A training batch is split into many smaller micro-batches. The first GPU stage performs a forward pass on a micro-batch and sends the resulting activations to the next stage. As the second stage begins its work, the first stage starts on the next micro-batch. This creates a staggered, parallel execution flow. The backward pass follows the same pattern in reverse, with gradients being passed back down the pipeline. Each stage accumulates its gradients locally across all micro-batches. The main drawback is the 'pipeline bubble,' an initial ramp-up and final ramp-down period where not all GPUs are active, which can hurt efficiency.

When to use it

Use pipeline parallelism when a model is too large to fit into the memory of a single accelerator, even after applying other memory-saving techniques. It is essential for training state-of-the-art foundation models with hundreds of billions of parameters. It is often used in a hybrid approach with data parallelism, where groups of GPUs form a pipeline, and these pipelines are replicated for data parallelism.

When not to use it

Avoid it for models that can fit on a single GPU. The communication overhead of passing activations between stages and the inefficiency from pipeline bubbles can make it slower than standard data parallelism for smaller models. It also requires a model architecture that can be cleanly split into sequential blocks.

One canonical example

The DeepSpeed library simplifies this. A torch.nn.Sequential model can be converted into a pipeline-parallel model with one line. By wrapping the sequential model in deepspeed.pipe.PipelineModule and specifying num_stages, DeepSpeed automatically partitions the model's layers and distributes them to the specified number of GPUs.