Hybrid parallelism for large-model training

WHAT THIS TESTS Whether you understand why pure data parallelism stops scaling for very large models and how mixing in model-partitioning strategies redistributes the communication burden.

A GOOD ANSWER COVERS In pure data parallelism, every GPU holds a full copy of the model and processes a different shard of the batch, then all replicas synchronize gradients with an all-reduce each step. The all-reduce volume scales with the parameter count, so for huge models the gradient synchronization dominates and adding more replicas yields diminishing returns. Tensor parallelism attacks this by sharding individual layers, for example splitting a large matmul column-wise and row-wise across GPUs; this introduces frequent all-reduce or all-gather operations within the forward and backward pass of a single layer, so it is bandwidth hungry and is kept inside a node over fast interconnect like NVLink. Pipeline parallelism splits the model into sequential stages placed on different GPUs, communicating activations and gradients between adjacent stages with point-to-point sends, and uses micro-batching to keep stages busy and shrink the pipeline bubble. Hybrid setups combine all three into a 2D or 3D mesh: tensor parallelism within a node, pipeline parallelism across nodes, and data parallelism over the resulting model replicas. Because each data-parallel replica now holds only a fraction of the parameters, the gradient all-reduce shrinks proportionally.

COMMON WRONG ANSWERS Believing that simply adding more data-parallel GPUs always speeds training ignores the growing all-reduce cost. Conflating tensor and pipeline parallelism, or claiming pipeline parallelism needs all-reduce rather than point-to-point sends, signals shallow understanding.

LIKELY FOLLOW-UPS What is the pipeline bubble and how do micro-batches reduce it? Why keep tensor parallelism within a node? How does ZeRO sharding differ from tensor parallelism?

ONE CONCRETE EXAMPLE Training a model too big for one GPU, you place tensor parallelism across 8 GPUs in a node over NVLink, chain 4 such nodes as pipeline stages over the network, then replicate that whole pipeline 16 times for data parallelism. Each replica now syncs only its shard of parameters, so the all-reduce that previously throttled scaling becomes a fraction of its former size.