Celery: Offloading Work from Your FastAPI App

Why it exists

Web servers are designed for short-lived HTTP request-response cycles. A long-running task, like processing a large file or sending thousands of emails, would block the server, leading to request timeouts and a poor user experience. Celery was created to solve this by moving heavy work out of the synchronous request flow.

The mental model

Think of your FastAPI application as a restaurant's front-of-house staff taking orders. Celery is the kitchen crew. The front staff doesn't cook; they write the order on a ticket (a "task") and put it on a message queue (the "message broker," like Redis or RabbitMQ). The kitchen (the "Celery worker") picks up the ticket, prepares the dish, and can report back when it's ready. This keeps the front staff free to serve more customers without waiting for the food to be cooked.

How it works

When an action in your FastAPI app needs to be run in the background, you call a special Celery function, known as a task. This function call and its arguments are serialized into a message and sent to the message broker. One or more separate Celery worker processes are constantly listening to this broker's queue. When a worker sees a new task, it picks it up, executes the function, and can optionally store the return value in a result backend for later retrieval.

When to use it

Use Celery for any task that is too slow to run inside an HTTP request. Common use cases include processing images or PDFs, sending bulk emails, generating large data exports, performing system backups, or running computationally intensive machine learning models. It is also ideal for periodic or scheduled jobs, like nightly data aggregation.

When not to use it

Avoid Celery for tasks that are fast and whose results are needed immediately within the same HTTP response. The overhead of creating a task, sending it to the broker, and having a worker pick it up adds latency. For simple, non-blocking I/O within a request, FastAPI's built-in BackgroundTasks feature is a lighter-weight solution that doesn't require extra infrastructure like a message broker.

One canonical example

A user uploads a large video file to a FastAPI endpoint. The endpoint validates the file type, saves it to storage, and immediately returns a 202 Accepted response to the user. In the background, it calls a Celery task like process_video.delay(video_id). A Celery worker then picks up this task to transcode the video into different formats and resolutions, a process that might take several minutes, without ever blocking the web server or making the user wait.