Elastic Training: Training Models on Unreliable Hardware
Elastic Training lets ML training jobs survive worker nodes being added or removed mid-run. It's like a construction crew that adapts to a changing number of workers, making it ideal for training large models on cheap but unreliable cloud spot instances.
WHY IT EXISTS: Training large models is expensive and time-consuming. Using cheaper, preemptible cloud "spot" instances can slash costs, but these instances can be taken away at any moment, crashing a standard training job. Elastic training was created to make these long-running jobs resilient to such failures.
THE MENTAL MODEL: Think of it as a self-managing construction crew. If some workers leave for the day or new ones show up, the foreman doesn't halt the entire project. Instead, they re-distribute tasks among the available crew, and the build continues from where it left off.
HOW IT WORKS: Elastic training relies on a framework (like PyTorch Elastic) that manages a dynamic group of worker nodes. A central agent, the "rendezvous backend," keeps track of which workers are active. When a worker drops, the remaining workers coordinate to load the last saved checkpoint and redistribute the workload. When a new worker joins, it's assigned a portion of the work, allowing the job to scale up. This requires frequent, efficient checkpointing of the model state.
WHEN TO USE IT: For long-running, distributed training jobs (hours, days, or weeks) on infrastructure where the number of nodes is not guaranteed. It's the standard for cost-effective training on cloud spot instances or in shared clusters with fluctuating resource availability.
WHEN NOT TO USE IT: It's overkill for small models or short training jobs where the risk of failure is low and the job can be easily restarted. If you are training on a single, stable machine or a fixed-size cluster of reserved instances, the added complexity is unnecessary.
ONE CANONICAL EXAMPLE: A team uses PyTorch's torchrun to train a large language model on a Kubernetes cluster that autoscales based on spot instance availability. The training job is configured with a minimum of 8 GPUs and a maximum of 64. As spot instances become available, the job scales up to use them, and as they are preempted, the job scales down without failing, continuing from the last checkpoint.
Read the original → iterate.ai
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.