Robust checkpointing strategy for multi-day training jobs and seamless resumption

WHAT THIS TESTS: This tests whether you understand that at frontier scale, hardware failure is a statistical certainty, not a risk to mitigate. With a node MTBF of roughly ten years, a 10,000-GPU cluster expects a failure every nine hours. The interviewer wants to see that you treat checkpoint and recovery reliability as core engineering, not infrastructure overhead. They are looking for awareness of the full stack: write throughput, storage durability, atomic semantics, corruption detection, and operational runbooks.

A GOOD ANSWER COVERS: First, async non-blocking checkpoint writes. The training loop should not stall while waiting for storage IO. Use async libraries like PyTorch DCP or TorchSnapshot, finalize atomically with a marker file or rename so partial writes are never picked up as valid. Second, MTBF-driven cadence. Derive checkpoint frequency from cluster size and expected failure rate to bound lost work, typically every tens of minutes to an hour, not once a day. Third, tiered storage economics. Hot tier can be fast node-local NVMe or burst buffers for async staging; warm tier is a parallel filesystem like Lustre or WekaFS for recent checkpoints; cold tier is object storage like S3 or GCS for long-term retention and cross-region durability. Fourth, integrity and recovery discipline. Include checksums or hashes to catch silent data corruption from cosmic rays or NIC drops. Practice recovery drills before you need them, and make the training script idempotent so resuming from the latest valid checkpoint is automatic and rank-safe.

COMMON WRONG ANSWERS: Proposing synchronous blocking checkpoints that pause all GPUs while rank zero writes to a shared filesystem. This destroys throughput and does not scale past a single node. Another red flag is ignoring atomic finalization, which leaves the system vulnerable to reading partially written checkpoints after a mid-write node failure. Suggesting daily checkpoints on a large cluster implies accepting a full day of lost compute on every failure, which is economically untenable at scale. Failing to mention silent corruption or checksums signals inexperience with production training.

LIKELY FOLLOW-UPS: How would you handle a straggler rank that slows down the collective checkpoint finalization? What is your strategy for resharding a checkpoint when the replacement node has a different GPU count or topology? How do you bound the storage cost when each checkpoint is tens of terabytes? Walk me through the exact recovery runbook when a node drops at step N plus one.

ONE CONCRETE EXAMPLE: Consider a 100,000-H100 training run. A synchronous write of a 500GB sharded checkpoint to a shared filesystem could take ten minutes and stall the entire cluster. Instead, use async DCP writes to local NVMe, then background migrate to a warm parallel filesystem, and finally archive to S3 with multipart upload. Checkpoint every thirty minutes based on the cluster MTBF. On failure, the orchestrator kills the unhealthy node, schedules a replacement, and the training script automatically resumes from the latest atomic checkpoint verified by checksum. Without this pipeline, a single NIC drop could cost a week of progress.