CI/CD Pipelines: How Stages and Jobs Orchestrate Work

WHY IT EXISTS Automating the process of building, testing, and deploying code without a clear structure would be chaotic and difficult to manage. Pipelines introduce order, control, and visibility by defining a predictable path from code commit to production release, ensuring quality at each step.

THE MENTAL MODEL Imagine a factory assembly line for your software. A pipeline is the entire line. Stages are the distinct, sequential stations along the line, like 'Assembly', then 'Quality Control', then 'Packaging'. Jobs are the individual workers or robots performing tasks in parallel at each station. The product cannot move to 'Quality Control' until all 'Assembly' tasks are complete.

HOW IT WORKS Pipelines are typically defined in a YAML file (like .gitlab-ci.yml). The configuration specifies a series of stages that run in a specific order. Each stage contains one or more jobs. All jobs within a single stage are executed in parallel by runners. The pipeline only proceeds to the next stage if all jobs in the current stage succeed. If any job fails, the pipeline stops by default, preventing flawed code from moving forward.

WHEN TO USE IT This stage-and-job model is the foundation of most CI/CD platforms. It is perfect for standard workflows where steps have clear dependencies: you must build code before you can test it, and you must pass all tests before you can deploy it. It provides a simple, robust, and easy-to-understand structure for automated software delivery.

WHEN NOT TO USE IT The strict sequential nature of stages can create bottlenecks. If a fast, independent job is in a later stage, it must wait for all jobs in earlier stages to finish, even if there's no direct dependency. For complex or large-scale projects, this can be inefficient. In such cases, it's better to define explicit dependencies between jobs (using features like GitLab's needs keyword) to create a more flexible execution graph (a DAG) that maximizes parallelism.

ONE CANONICAL EXAMPLE A typical pipeline has three stages. First, a 'build' stage with a 'compile' job. Second, a 'test' stage with two parallel jobs, 'run-unit-tests' and 'run-integration-tests'. This stage only starts if 'compile' succeeds. Third, a 'deploy' stage with a 'deploy-to-production' job, which only runs if both test jobs pass. This ensures code is never deployed unless it builds and passes all tests.