Dynamic Fan-out/Fan-in Pipelines

Why it exists

Static data pipelines force you to decide how many parallel branches you need before the pipeline runs. In production ML systems, the amount of work is rarely constant. One day you might have three customer segments to retrain on, the next day thirty. Hard-coding branch counts wastes compute when load is low and breaks when load is high. Dynamic fan-out/fan-in exists so that the pipeline can match its width to the actual workload discovered at runtime.

The mental model

Think of a distribution center that does not know how many trucks will arrive each morning. Instead of building fifty fixed loading docks and leaving most empty, the center opens exactly one dock per truck that shows up, processes them simultaneously, and then closes them all once every parcel is back on a single outbound conveyor. The pipeline behaves the same way: it expands to fit the job, then contracts to deliver a unified result.

How it works

An initial task inspects the outside world, perhaps listing files in a landing bucket or querying a database for active experiment IDs. It returns a collection, such as a list of file paths or configuration dictionaries. The orchestrator then generates one parallel task per item in that collection, passing each item as input to its own branch. These branches run concurrently. A special fan-in task waits for the entire collection to complete and receives the outputs as an aggregated list or merged artifact. The key difference from static DAGs is that the graph structure is materialized at runtime, not defined ahead of time.

When to use it

Use dynamic fan-out/fan-in whenever the number of parallel units depends on data you cannot predict before the pipeline starts. Common scenarios include processing a variable number of daily partition files, running hyperparameter sweeps where the grid is generated by an upstream analysis, retraining one model per region or customer segment discovered in a metadata table, and embarrassingly parallel feature engineering across shards.

When not to use it

Avoid this pattern when the branch count can explode into the thousands, because most orchestrators store task metadata in a database that bloats with excessive task instances. Do not use it if the parallel branches need complex cross-communication or strict ordering, since the pattern assumes independence. Also skip it for very short tasks where scheduler overhead dominates runtime; in those cases, batch the work into fewer coarse branches instead.

One canonical example

A daily recommendation model pipeline wakes up and scans an S3 prefix for new user behavior logs. It discovers twelve parquet files today. The pipeline fans out to twelve parallel feature engineering tasks, each consuming one file and writing a processed shard. A fan-in task then collects all twelve shard paths, trains a single gradient boosted model on the unified dataset, and promotes the model to staging. Tomorrow the same pipeline might find three files or forty, and it will automatically adjust its width without any code changes.