Backpressure: Don't Drown Your Node.js Streams

WHY IT EXISTS To prevent system instability when a data source produces data faster than a destination can consume it. Without flow control, the excess data must be buffered in memory, which can quickly lead to memory exhaustion and crash the application, especially in a single-threaded environment like Node.js.

THE MENTAL MODEL Imagine a factory conveyor belt. One worker (the producer) places items on the belt, and another worker at the end (the consumer) processes them. If the producer is faster, items pile up and fall off. Backpressure is the consumer signaling the producer to pause until the backlog is cleared. In Node.js streams, this prevents a fast file read from overwhelming a slow network client.

HOW IT WORKS Node.js streams use a simple signaling system based on a buffer's high-water mark. When a writable stream's internal buffer is full, its .write() method returns false. This is the backpressure signal. A well-behaved readable stream will see this false return value and stop pushing data. Once the writable stream has processed some of its buffered data and the buffer level drops below the high-water mark, it emits a 'drain' event. This signals the readable stream that it's safe to resume pushing data. The built-in .pipe() method automates this entire negotiation.

WHEN TO USE IT Backpressure is a core feature of streams and should be respected in any stream-based data transfer where the source and destination operate at different speeds. This is common in file I/O piped to network I/O, or when performing complex, time-consuming transformations on each data chunk. Using .pipe() is the easiest way to ensure it's handled correctly.

WHEN NOT TO USE IT It's rarely a good idea to ignore backpressure. If you are certain that your data set is small enough to fit entirely in memory and the consumer is fast enough, you might read the whole source at once. However, using streams with their built-in backpressure handling is the more robust and scalable approach for most real-world applications.

ONE CANONICAL EXAMPLE Serving a large video file with http.createServer. A naive implementation might use fs.readFile and then res.end(data), which loads the entire multi-gigabyte file into RAM. The correct way is fs.createReadStream('large-video.mp4').pipe(res). Here, .pipe() automatically handles backpressure, ensuring that Node.js only keeps a small chunk of the file in memory at any given time, reading from the disk only as fast as the client can download the data over the network.