How do you adapt ML training for spot instance interruptions?

WHAT THIS TESTS: This question tests whether you can bridge ML experimentation with production-grade distributed systems. The interviewer cares if you understand that cost optimization via spot instances is only viable when you decouple compute reliability from training progress. They want to see awareness of preemption signals, durable persistence, and idempotent orchestration.

A GOOD ANSWER COVERS: First, granular checkpointing. Instead of saving at epoch boundaries, write model weights, optimizer state, and RNG seeds to object storage like S3 every N steps. Second, signal handling. The training script or its wrapper must catch SIGTERM, which AWS sends up to two minutes before interruption, to flush buffers and upload a final emergency checkpoint. Third, orchestration logic. The scheduler should detect failure, spin up a replacement spot instance, and resume training from the latest valid checkpoint rather than treating the job as failed. Fourth, overhead management. You must balance checkpoint frequency against upload latency and storage costs; for large models, use asynchronous checkpointing or sharded state dicts to avoid blocking the GPU. Fifth, validation and cleanup. On resume, verify checkpoint integrity with checksums and garbage collect stale checkpoints to avoid storage bloat.

COMMON WRONG ANSWERS: A major red flag is checkpointing only to local ephemeral storage like EBS or NVMe. When the spot instance terminates, that state vanishes. Another mistake is ignoring the preemption signal entirely and relying solely on periodic checkpoints, which can lose hours of compute. Some candidates suggest complex distributed consensus or leader election, which is over-engineering for a deterministic training job. Finally, proposing checkpointing every single step without discussing I/O overhead shows lack of practical systems thinking.

LIKELY FOLLOW-UPS: How would you handle a spot interruption during a distributed multi-node training run? What is your strategy if the checkpoint itself is corrupted? How do you cost-compare spot savings against the overhead of frequent S3 uploads? Would you use a managed training service, and what are its trade-offs?

ONE CONCRETE EXAMPLE: Imagine a ResNet-50 training job on four spot GPUs. You modify the PyTorch loop to save checkpoint shards to S3 every 500 steps using torch.save. A sidecar process polls the instance termination notice endpoint. When a termination is announced, it sends SIGTERM to the training process, which triggers an immediate synchronous checkpoint upload. Your orchestrator, perhaps Airflow or a custom controller, sees the node disappear, requests a new spot instance, and restarts the job pointing to the latest S3 prefix. The job resumes from step 24700 instead of restarting from epoch zero.