Directed Acyclic Graph (DAG) for Workflows

WHY IT EXISTS Real pipelines are not single linear scripts; they have steps that depend on others and steps that can run at the same time. You need a structure that captures those dependencies precisely so a scheduler can run things in the right order, parallelize what is independent, and recover from partial failures. The DAG is that structure.

THE MENTAL MODEL Picture each task as a node and each dependency as an arrow pointing from a prerequisite to a dependent task. Directed means arrows have a direction; acyclic means you can never follow arrows in a loop back to where you started. That absence of cycles is what makes a valid execution order, a topological sort, always exist.

HOW IT WORKS The orchestrator reads the graph, computes a topological ordering, and dispatches any task whose upstream dependencies have all succeeded. Tasks with no dependency relationship between them run concurrently. If a task fails, only it and its downstream descendants are affected; with idempotent tasks the engine can retry that subtree without rerunning the whole pipeline. Each node typically declares its inputs, outputs, and retry policy.

WHEN IT MATTERS DAGs matter whenever workflows branch, fan out, or must be partially re-run, which is nearly every ETL and ML training pipeline. They also make lineage and scheduling explicit and auditable.

ONE CONCRETE EXAMPLE In Airflow, an ML pipeline DAG has ingest feeding both validate and a checksum task, validate feeding transform, transform feeding train, and train feeding evaluate then deploy. Ingest and an unrelated config-fetch task run in parallel. If train fails, evaluate and deploy are skipped while upstream successful tasks are not rerun on retry.