Resilient stateful batch on Spot Instances

WHAT THIS TESTS This probes whether you can engineer resilience around compute that can disappear with little warning, turning a cheap but volatile resource into a dependable platform for stateful work.

A GOOD ANSWER COVERS The core principle is to externalize state so no instance is irreplaceable. Periodically checkpoint progress, such as the last processed offset or intermediate results, to durable storage like S3 or a managed database, so a replacement instance resumes from the checkpoint instead of restarting from zero. React to lifecycle signals: AWS emits a Spot interruption notice roughly two minutes before reclamation and a rebalance recommendation earlier when risk rises, so a handler should catch these, flush a final checkpoint, deregister from any load balancer or queue, and exit cleanly. Diversify the capacity pool across many instance types, sizes, and Availability Zones so a shortage in one pool does not reclaim everything at once, and run a baseline on On-Demand or reservations so the job always makes progress. Design each unit of work to be idempotent and chunked, so reprocessing after an interruption produces no duplicates or corruption. Use a managed orchestrator like Spot Fleet, EC2 Auto Scaling with capacity-optimized allocation, or AWS Batch to automate replacement.

COMMON WRONG ANSWERS Storing working state only in instance memory or local disk, which is lost on reclamation. Ignoring the interruption and rebalance notifications and letting work die mid-chunk. Picking a single instance type, concentrating interruption risk. Assuming the two-minute window is enough to finish a long task rather than just to checkpoint.

LIKELY FOLLOW-UPS How small should checkpoints be. How do you guarantee exactly-once effects. How do you fall back when Spot capacity is unavailable.

ONE CONCRETE EXAMPLE A genomics pipeline splits work into idempotent chunks, writes each completed chunk and a progress marker to S3, and runs on a diversified Spot Fleet. An interruption handler flushes the current chunk's checkpoint within the two-minute notice, and a replacement instance picks up from the last marker, so cost drops sharply with no lost results.