Difference between data and model parallelism, and when to prefer each

WHAT THIS TESTS: Whether you understand the axis along which work is distributed in distributed training. Data parallelism and model parallelism are not interchangeable optimizations; they solve different constraints. The interviewer wants to see that you know what is replicated, what is partitioned, and how that maps to hardware limitations like GPU memory versus data throughput.

A GOOD ANSWER COVERS: First, define data parallelism as replicating the entire model on each worker and splitting the global batch across workers, citing PyTorch Distributed Data Parallel or Fully Sharded Data Parallel as examples. Second, define model parallelism as splitting the model itself across workers, citing Tensor Parallel or Pipeline Parallelism as examples. Third, give a clear decision rule: use data parallelism when the model fits comfortably in a single GPU and you need to train on more data faster; use model parallelism when a single layer or the full model exceeds GPU memory. Fourth, mention that hybrid strategies exist and that real systems often combine both.

COMMON WRONG ANSWERS: Calling FSDP model parallelism is a frequent mistake; FSDP is still a data parallel strategy. Another red flag is saying data parallelism is only for small models; it is used at massive scale when the model fits. Ignoring communication overhead is also weak; both strategies introduce network traffic that can bottleneck training. Finally, giving a vague scenario without memory constraints suggests you have not operated these systems in practice.

LIKELY FOLLOW-UPS: How does FSDP differ from DDP and why is it still data parallelism? When would you combine Tensor Parallel with Pipeline Parallelism? How do you handle idle time in pipeline parallelism? What changes in the backward pass for model parallelism compared to data parallelism? How do you partition data when using model parallelism so that every worker still sees the same input batch?

ONE CONCRETE EXAMPLE: Imagine a ten billion parameter transformer where a single transformer layer consumes twenty gigabytes of parameters in fp32. A single A100 with eighty gigabytes cannot hold the full model, let alone activations. You would choose Tensor Parallel to shard individual matrix multiplications within a layer across devices, or Pipeline Parallel to place different layers on different GPUs. Conversely, if you have a one hundred million parameter ResNet that fits in four gigabytes, you would replicate it on eight A100s with DDP and feed each GPU a distinct microbatch, scaling throughput nearly linearly.

Source: PyTorch Distributed Overview