Python's Asyncio Subprocesses: Non-Blocking Shell Commands

Why it exists

Standard subprocess calls are blocking. In an async application like a web server, a blocking call freezes the entire event loop, preventing it from handling other requests. Asynchronous subprocesses solve this by integrating external process management directly into the asyncio event loop, allowing the app to remain responsive.

The mental model

Think of it like delegating a task to an intern and telling them to report back when they're done, rather than doing it yourself while all your other work piles up. Your main application (the event loop) hands off the command to the operating system and can continue handling other tasks, like serving web requests, until the subprocess signals it has finished.

How it works

You use await asyncio.create_subprocess_exec() to run a program with arguments or await asyncio.create_subprocess_shell() for a full shell command. These functions are async and return a Process object without waiting for the command to finish. To interact with the process, you can pipe its I/O streams using asyncio.subprocess.PIPE. To get the final result and wait for completion, you await the proc.communicate() method, which returns the stdout and stderr as byte strings. The exit code is then available in proc.returncode.

When to use it

Use this when you need to run external command-line tools from within an asyncio-based application, such as a FastAPI or aiohttp web server. It's perfect for tasks like video transcoding with ffmpeg, running a git command, or executing a data processing script, all without blocking the server's main thread. It's also powerful for running multiple independent commands in parallel using asyncio.gather.

When not to use it

Avoid this for simple, synchronous scripts where the standard subprocess module is a better fit. The biggest footgun is using create_subprocess_shell with any string that includes user-provided input, which opens a major shell injection vulnerability. create_subprocess_exec is almost always the safer choice as it does not invoke a system shell. If you must use the shell, meticulously sanitize all inputs with shlex.quote().

One canonical example

To run a command and capture its output, you first create the process with proc = await asyncio.create_subprocess_shell('ls -l', stdout=asyncio.subprocess.PIPE). This starts the command but your code continues. Later, when you need the result, you call stdout, stderr = await proc.communicate(). The stdout variable will now contain the output of the ls command as a bytes object, which you can decode to a string. Your event loop is free to do other work between the create and communicate calls.