Offload long-running tasks from web requests

WHAT THIS TESTS This is an async-architecture design question. The interviewer wants the producer-queue-consumer pattern and an explanation of how it stays reliable across restarts and scaling.

A GOOD ANSWER COVERS A web request must return quickly, so the heavy work cannot run inline; HTTP timeouts and tied-up workers would result. The pattern is to decouple submission from execution with a message queue. When the user requests a report, the web tier validates input, places a job message on a durable queue such as Amazon SQS, Google Pub/Sub, or a broker like Redis or RabbitMQ behind Celery, and immediately responds with 202 Accepted plus a job id. A separate pool of worker instances or processes consumes messages from the queue, generates the report, and writes the finished file to object storage, then marks the job complete. The client learns the result by polling a status endpoint, receiving a webhook, or a push notification. The queue is the key reliability mechanism: it persists the job so a worker crash does not lose it, redelivers on failure with retries, supports a dead-letter queue for poison messages, and lets the web tier and worker tier scale independently.

COMMON WRONG ANSWERS Running the report inside the request handler, causing timeouts and exhausted web workers. Spawning a background thread in the web process, which dies when the ephemeral instance restarts or redeploys, silently losing the job. No retry or visibility into status. Coupling worker scaling to web scaling.

LIKELY FOLLOW-UPS How do you guarantee at-least-once versus exactly-once processing and handle idempotency? What is a dead-letter queue? How do workers autoscale on queue depth? How does the client get notified, and how do you handle very long jobs?

ONE CONCRETE EXAMPLE A user clicks Generate Report. The API enqueues {reportId, params} to SQS and returns 202 with reportId. Worker containers pulling from SQS render the PDF, upload it to S3, and set status=done in the database. The frontend polls GET /reports/{id} until it returns a signed S3 download URL. If a worker crashes mid-job the message reappears after the visibility timeout and another worker retries it.