Cost-effective fault-tolerant batch processing
Source: interviewadvanced
WHAT IT TESTS: batch architecture and cost-fault trade-offs. OUTLINE: Spot fleets across types, a durable work queue, idempotent checkpointed tasks, retries.
WHAT IT TESTS: whether you combine cheap interruptible compute with a design that survives interruptions. ANSWER OUTLINE: run on a diversified Spot fleet across many instance types and AZs for capacity and price, pull work from a durable queue, make tasks idempotent and checkpoint progress to object storage, and rely on visibility timeouts and retries so reclaimed work is reprocessed. Use a managed batch service to orchestrate.
Read the original → interview
- #batch-processing
- #spot-instances
- #fault-tolerance
- #queue
- #idempotency
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.