Cost-effective fault-tolerant batch processing

WHAT THIS TESTS The interviewer wants an end-to-end design that exploits the cheapest compute while staying correct under frequent interruption.

A GOOD ANSWER COVERS Choose Spot or preemptible instances for the deep discount, and diversify the fleet across many instance types and availability zones so the scheduler can always find capacity and avoid correlated reclamation. Decouple work distribution with a durable queue: workers pull a task, process it, and acknowledge completion. Design every task to be idempotent so reprocessing after an interruption is safe, and checkpoint partial progress to object storage for long tasks. Use the queue's visibility timeout so an unacknowledged task automatically becomes available again when its worker is reclaimed, and cap retries with a dead-letter queue for poison messages. A managed batch service can orchestrate the fleet, retries, and scaling. Scale workers on queue depth so the fleet grows with backlog and drains to zero when idle.

COMMON WRONG ANSWERS Using a single Spot instance type, inviting mass reclamation. Pushing tasks to workers instead of pulling, so an interrupted worker silently drops its task. Non-idempotent tasks that double-charge or duplicate output on retry. No checkpointing, so long jobs restart from zero. Ignoring poison messages that retry forever.

LIKELY FOLLOW-UPS How does the visibility timeout guarantee at-least-once processing. How do you make a task idempotent. How do you mix a small On-Demand base with a Spot bulk for reliability. How do you detect and quarantine poison messages.

ONE CONCRETE EXAMPLE A genomics pipeline must align millions of samples overnight. Tasks land in a queue; a managed batch service launches a diversified Spot fleet that scales on queue depth. Each worker checkpoints alignment progress to object storage and writes idempotent output keyed by sample ID. When Spot reclaims a node mid-task, the visibility timeout returns the message, another worker resumes from the last checkpoint, and total cost is a fraction of On-Demand with no lost or duplicated results.