Diagnosing poor distributed training scaling

WHAT THIS TESTS Whether you understand the real limits of data-parallel scaling and can name concrete tools to localize the bottleneck.

A GOOD ANSWER COVERS The usual suspect is communication overhead. In data-parallel training, gradients are synchronized via all-reduce every step; as you add workers, communication volume and synchronization cost grow and can outpace compute, especially over slow interconnects. Check whether GPUs talk over PCIe versus NVLink within a node and Ethernet versus InfiniBand across nodes. The second suspect is the input pipeline: if data loading, decoding, or augmentation cannot keep up, GPUs starve and sit idle. Third is load imbalance and stragglers, where one slow worker or uneven shard stalls the synchronous step. Fourth is too small a per-GPU batch, which underutilizes each GPU so fixed overheads dominate. Also consider gradient size (use mixed precision and gradient compression or bucketing) and CPU or memory limits.

HOW TO PROFILE Use the PyTorch profiler with the trace viewer to see compute versus communication time, NVIDIA Nsight Systems for a system-wide timeline, nvidia-smi and dcgm for GPU utilization, and built-in NCCL logging to inspect all-reduce timing. Low GPU utilization with high communication time points to the interconnect or comm pattern; low utilization with idle gaps points to data loading.

COMMON WRONG ANSWERS Assuming scaling is automatically linear, blaming the GPUs and simply adding more nodes, or never profiling to separate compute from communication.

ONE CONCRETE EXAMPLE The profiler shows GPUs at thirty percent utilization with large all-reduce bars. Moving from Ethernet to InfiniBand, enabling NCCL bucketing, and overlapping backward pass with gradient communication restores near-linear scaling, while increasing per-GPU batch size raises utilization further.