Why is FastAPI BackgroundTasks poor for multi-minute PDF generation?

WHAT THIS TESTS: This question probes whether you understand the scope and limitations of FastAPI's BackgroundTasks. According to the FastAPI documentation, BackgroundTasks is intended for operations that need to happen after a request but that the client does not need to wait for, such as sending email notifications that take several seconds. It is not designed for heavy background computation that runs for minutes. The interviewer wants to see that you recognize same-process, non-persistent execution and can design a decoupled, durable system.

A GOOD ANSWER COVERS: First, explain that BackgroundTasks runs inside the same Starlette process and typically uses the same event loop or thread pool. A multi-minute PDF generation job would block or starve a worker for the entire duration, reducing throughput and risking timeouts. Second, note that there is no persistence: if the server restarts or the process crashes, the task disappears with no retry mechanism. Third, propose a dedicated task queue architecture. Name specific technologies such as Celery, ARQ, or RQ backed by Redis or RabbitMQ. Describe the key components: a message broker to hold jobs, separate worker processes to consume them, a result backend to track state, and object storage like S3 for the generated PDF. Fourth, describe the API flow: the endpoint validates the request, enqueues a job, and immediately returns HTTP 202 Accepted with a job ID. The client then polls a status endpoint or receives a webhook when the worker finishes.

COMMON WRONG ANSWERS: A major red flag is suggesting that you simply add more Uvicorn workers or horizontally scale FastAPI containers to handle the load. This does not solve durability, retries, or visibility into job state. Another weak answer is proposing asyncio.create_task or threading without a persistent queue, which still dies on process restart. Saying BackgroundTasks is fine because it runs after the response is also wrong for minute-scale work.

LIKELY FOLLOW-UPS: The interviewer may ask how you would handle task retries and dead-letter queues for failed PDF generations. They might ask about idempotency if the client submits the same report twice. You could also be asked about backpressure: what happens if the queue grows faster than workers can process, and how you would scale workers independently of the API tier. Another follow-up is security: how do you ensure the poll endpoint only exposes the requesting user's jobs.

ONE CONCRETE EXAMPLE: Imagine a report service using Celery with Redis as the broker and backend. The FastAPI endpoint receives a request for a 10,000-row PDF, pushes a task to the reports queue, and returns 202 with job ID 42. A Celery worker on a separate instance picks up the task, renders the PDF into a temporary file, uploads it to S3 with a presigned URL, and marks the job as completed in Redis. The client polls GET /jobs/42 every two seconds; after three minutes it receives the S3 URL. If the worker dies midway, Celery automatically retries up to three times.