Tensor versus pipeline parallelism for large models

WHAT THIS TESTS The interviewer wants to know whether you can deploy a model that exceeds single-GPU memory and reason about the communication characteristics of each sharding scheme. This is core distributed-inference and training knowledge.

A GOOD ANSWER COVERS Tensor parallelism, also called intra-layer parallelism: split the weight matrices of individual layers, such as the attention and feed-forward projections, column-wise or row-wise across GPUs. Each GPU computes a partial result for the same layer, then the partials are combined with an all-reduce or all-gather every layer. This communicates frequently and at high volume, so it requires very fast interconnect like NVLink and is best confined to GPUs within a single node. Pipeline parallelism, inter-layer parallelism: assign contiguous groups of layers to different GPUs as pipeline stages; a batch flows stage to stage, with each GPU sending only the activations at stage boundaries, so communication is far lighter and tolerates slower cross-node links. The cost is pipeline bubbles, idle time while the pipeline fills and drains, mitigated by splitting batches into micro-batches. Decision: a 70B model in FP16 needs roughly 140 GB, exceeding one GPU. Use tensor parallelism across the GPUs inside a node where NVLink is fast, and pipeline parallelism across nodes where bandwidth is lower; for the largest models combine both, plus data parallelism for throughput. The guiding principle is to match the high-communication scheme to the fastest links.

COMMON WRONG ANSWERS Confusing either with data parallelism, which replicates the full model on each GPU and splits the batch, useless when the model itself does not fit. Saying pipeline parallelism communicates more than tensor parallelism. Ignoring the interconnect constraint that drives the choice.

LIKELY FOLLOW-UPS What are pipeline bubbles and how do micro-batches help. Why does TP need NVLink. How do TP, PP, and DP compose in 3D parallelism. Where does sequence or expert parallelism fit.

ONE CONCRETE EXAMPLE On two nodes of eight GPUs each, set tensor parallel degree eight within each node over NVLink and pipeline parallel degree two across the two nodes, fitting the 70B model while keeping the chatty all-reduce traffic on the fast intra-node fabric.