Explain data, tensor, and pipeline parallelism and hybrid training strategy

What's really being asked

Your ability to reason about distributed memory hierarchies and communication patterns at scale. Interviewers want to see that you understand why a single parallelism strategy fails for 100B+ parameter models and how to orchestrate multiple strategies based on hardware topology.

The full answer

First, data parallelism replicates the full model weights on every worker and shards the global batch; each worker computes gradients independently and synchronizes via all-reduce. This is simple but memory-bound because every GPU must hold the entire model. Second, tensor parallelism splits individual layers or matrices across devices within the same layer; workers exchange intermediate activations via all-reduce or all-gather inside the forward and backward passes. This reduces per-device memory but introduces high-bandwidth communication that is best kept inside a single node. Third, pipeline parallelism splits the model into sequential stages across devices; each stage holds a disjoint subset of layers and passes activations forward and gradients backward via point-to-point communication. This is sensitive to pipeline bubble and load imbalance. Fourth, a hybrid strategy composes these in a 3D mesh. A canonical pattern is tensor parallelism within a node to exploit NVLink bandwidth, pipeline parallelism across nodes in the same rack, and data parallelism across the largest dimension to scale batch size. You should mention that tensor parallelism is usually limited to the size of a single node because of all-reduce overhead, pipeline parallelism is bounded by the number of layers and bubble fraction, and data parallelism is bounded by the global batch size and gradient synchronization cost.

The mistakes people make

Saying data parallelism alone works if you just add more GPUs. Ignoring that tensor parallelism requires all-reduce on every layer and therefore saturates inter-node bandwidth. Describing pipeline parallelism without mentioning bubble overhead or the need for micro-batching to keep the pipeline full. Proposing a hybrid layout without respecting the physical network topology, such as placing tensor parallelism across slow Ethernet links. Confusing tensor parallelism with model parallelism in general without specifying the intra-layer sharding mechanism.

What usually comes next

How does ZeRO or Fully Sharded Data Parallel change this picture? What happens to activation memory under pipeline parallelism with micro-batching? How do you handle load balancing when layer compute is not uniform? What is the optimal micro-batch size to minimize bubble time? How would you adapt this strategy for inference instead of training?

A concrete example

Training a 175B parameter model on a cluster of 8-GPU nodes. You might place tensor parallelism across the 8 GPUs in a node so each shard holds roughly 22B parameters. Then you chain 16 such nodes in a pipeline, giving each pipeline stage about 11B parameters. Finally you replicate that pipeline 4 times with data parallelism to reach a global batch of 2048 samples. The result uses 512 GPUs, keeps tensor all-reduce inside NVLink domains, limits pipeline bubbles to a small fraction of total time, and scales batch processing without requiring any single GPU to hold more than its shard.