Backpressure: Slow the Producer or Crash

WHY IT EXISTS: Data streaming systems move records through a pipeline of producers, queues, and consumers. If a downstream stage slows down due to a garbage collection pause, a slow database query, or a traffic spike, records accumulate in buffers. Without a mechanism to stop or slow the upstream, memory grows unbounded until the process crashes or the kernel kills it. Backpressure exists to prevent this collapse by making the flow rate match the capacity of the slowest stage.

THE MENTAL MODEL: Think of a kitchen assembly line where the oven can only bake ten pizzas per hour but the prep station can make fifty. If prep keeps working at full speed, pizzas pile up on the counter, block walkways, and eventually the whole kitchen shuts down. Backpressure is the head chef telling prep to slow down until the oven catches up. In software, the counter is memory and the kitchen shutdown is an out-of-memory kill.

HOW IT WORKS: A consumer signals that its buffer is full or that its current load is above a threshold. The streaming runtime or protocol propagates this signal upstream, often by blocking writes, returning a not-ready status, or reducing a TCP window size. The producer then pauses or reduces its send rate until the consumer signals readiness again. In reactive streams, this is formalized through request-n semantics where a subscriber explicitly requests the next batch of items. In Kafka, the consumer's fetch lag and the producer's max-in-flight settings create an implicit form of backpressure.

WHEN TO USE IT: Use backpressure whenever a system has asymmetric processing speeds between stages and bounded memory is a constraint. It is essential in real-time analytics pipelines, log aggregation, and microservice RPC chains where a burst from a fast client can overwhelm a slow database. It is also critical in event-driven architectures where you cannot predict traffic spikes and would rather shed load through slowdown than crash.

WHEN NOT TO USE IT: Do not rely on backpressure if the business requirement is strict low latency and dropping data is preferable to slowing down. In some telemetry or metrics pipelines, it is better to sample or drop excess events than to stall the producer. Also avoid backpressure across organizational boundaries where the upstream team cannot tolerate a slowdown; in those cases, use a durable external queue like Kafka as a shock absorber rather than blocking the caller.

ONE CANONICAL EXAMPLE: A Flink job reads clickstream events from Kafka, applies a windowed aggregation, and writes results to a PostgreSQL database. During a flash sale, the database connection pool saturates and writes slow from ten milliseconds to two seconds. Flink's backpressure mechanism detects the bottleneck at the sink operator and propagates pressure back through the aggregation operator to the Kafka source, reducing the consumer's fetch rate. The pipeline stays up with stable memory usage instead of buffering millions of events and crashing the task managers.