Go's Worker Pool Pattern: Capping Concurrency

Why it exists

Spawning a goroutine for every single task, like processing a million files, is inefficient and can exhaust system resources. The worker pool pattern exists to limit the number of concurrently running tasks to a manageable, fixed number, ensuring stable performance under heavy load.

The mental model

Imagine a factory with a fixed number of assembly line stations (the workers). A conveyor belt (the jobs channel) brings parts to be assembled. Another conveyor belt (the results channel) takes finished products away. No matter how many parts are waiting, only a fixed number can be worked on at once, preventing the factory floor from being overwhelmed.

How it works

The pattern uses two channels: one for jobs and one for results. A controlling function, often main, first creates these channels. It then spawns a fixed number of worker goroutines. Each worker is a function that loops, reading a job from the jobs channel. After processing, it sends the outcome to the results channel. The controller sends all work items to the jobs channel and then closes it to signal that no more jobs are coming. Finally, the controller reads the expected number of results, which ensures all work is complete before the program exits.

When to use it

Use a worker pool when you have a large number of independent, similar tasks to perform concurrently, but you want to control the level of parallelism. This is ideal for processing items from a queue, making network requests to an external API with rate limits, or performing parallel computations on a multi-core CPU.

When not to use it

If you only have a handful of tasks, the overhead of setting up the pool might not be worth it; simply spawning a goroutine for each might be simpler. If tasks are not independent and need complex coordination, other concurrency patterns might be more appropriate. Also, if you don't need to collect results, a WaitGroup might be a simpler synchronization mechanism.

One canonical example

To process 5 jobs that each take 1 second, you can start 3 worker goroutines. You send the 5 jobs to a jobs channel and then close it. The first 3 workers will immediately pick up jobs 1, 2, and 3. As soon as one worker finishes (e.g., worker 1 finishes job 1), it picks up the next available job (job 4). With 3 workers, the 5 seconds of total work completes in about 2 seconds. The main goroutine waits by reading 5 items from the results channel.