Fault-tolerant checkpointing for thousand-GPU pre-training

WHAT THIS TESTS Whether you understand that correct resumption requires the full training state, and that at thousands of GPUs failures are routine so checkpointing must be cheap and automatic.

A GOOD ANSWER COVERS WHAT TO SAVE. A correct checkpoint is more than weights. You need optimizer states such as Adam first and second moments, the learning-rate scheduler step, the global step count, RNG states for every device so dropout and data shuffling continue deterministically, and the data loader position so you neither repeat nor skip samples. Missing any of these makes the resumed run diverge from an uninterrupted one. HOW TO WRITE IT. Use distributed, sharded checkpointing where each rank writes its own shard in parallel to durable storage, and make writes asynchronous so training continues while the checkpoint flushes, overlapping I/O with compute. Choose the interval by balancing checkpoint write cost against expected work lost per failure; with frequent failures, checkpoint more often. Keep at least one prior checkpoint and verify integrity so a crash mid-write cannot corrupt the only copy. RESUMPTION. A controller detects a dead or straggling node via heartbeats, replaces or drains it, then relaunches the job, which loads the latest valid checkpoint and resumes from the saved step, RNG, and data position. Elastic frameworks can rendezvous the surviving and replacement workers automatically.

COMMON WRONG ANSWERS Checkpointing only model weights. Synchronous writes that stall all GPUs. Forgetting RNG state, so randomness diverges. Forgetting data-loader position, causing repeated or skipped data. Keeping a single checkpoint that can be corrupted mid-write. Manual restart only.

LIKELY FOLLOW-UPS How do you pick the checkpoint interval mathematically? Why save RNG and data position for correctness? How do asynchronous and sharded checkpoints reduce overhead? How does elastic training handle a permanently lost node?

ONE CONCRETE EXAMPLE At step 50000 a GPU fails. Because checkpoints captured weights, Adam moments, scheduler step, per-rank RNG, and the data cursor at step 49500, the controller swaps in a spare node, the job reloads that snapshot, and training resumes at 49500 with identical randomness and the next unread data batch, losing only 500 steps and producing a run statistically identical to one that never failed.