How would you speed up slow single-GPU training?

WHAT THIS TESTS This checks that you distinguish scaling up from scaling out and understand the trade-offs of distributed deep learning, not just that bigger hardware is faster.

A GOOD ANSWER COVERS Two distinct strategies. First, vertical scaling: move to a larger instance with more or faster GPUs, enable mixed-precision training to use tensor cores, and increase batch size to keep the accelerators saturated. This is the simplest change and often the quickest win. Second, horizontal scaling with distributed training. Data parallelism replicates the model on each GPU, splits the batch, and synchronizes gradients each step via all-reduce; it suits most models. Model parallelism or pipeline parallelism shards a model that cannot fit in one GPU's memory across devices. A strong candidate notes the communication cost and that the learning rate usually must scale with the effective batch size.

COMMON WRONG ANSWERS Only suggesting a faster single GPU. Assuming N GPUs give exactly Nx speedup with no synchronization overhead. Confusing data parallelism with model parallelism. Ignoring data-pipeline bottlenecks where the GPU starves waiting on input.

LIKELY FOLLOW-UPS How does all-reduce work? When do you choose model over data parallelism? How do you keep the input pipeline from becoming the bottleneck? What about spot instances for cost?

ONE CONCRETE EXAMPLE On Azure ML or SageMaker you launch a cluster of GPU nodes and run PyTorch DistributedDataParallel with NCCL all-reduce over high-bandwidth interconnect. You enable automatic mixed precision, scale the learning rate with a warmup, and prefetch data with multiple workers so the GPUs stay busy. For a model too large for one device, you instead shard layers with pipeline parallelism such as DeepSpeed or Megatron.