Dynamic Pipelines: Parent-Child vs. Multi-Project

WHY IT EXISTS Monolithic CI/CD configurations become slow, rigid, and hard to manage as a project grows. Instead of a single, giant pipeline that runs everything, dynamic pipelines allow you to break the work into smaller, independent, and conditional workflows that run only when needed.

THE MENTAL MODEL Think of a main CI pipeline as a manager. Instead of doing all the work itself, it delegates tasks to specialized workers. Based on conditions like which files changed, the manager (the upstream pipeline) triggers one or more workers (downstream pipelines) to perform a specific job. This is more efficient than having one worker that knows how to do everything.

HOW IT WORKS A job in an upstream pipeline's .gitlab-ci.yml uses the trigger keyword to start a downstream pipeline. There are two main types. First, parent-child pipelines, for triggering another pipeline within the same project using trigger: include: path/to/child.yml. Second, multi-project pipelines, for triggering a pipeline in a different project using trigger: project: group/other-project. The downstream pipeline runs independently, but you can pass variables to it. By default, the trigger job succeeds as soon as the downstream pipeline starts, not when it finishes.

WHEN TO USE IT Use parent-child pipelines for complex logic within a single project, especially monorepos. A parent pipeline can detect which service's code has changed and dynamically trigger a child pipeline to test only that service. Use multi-project pipelines to coordinate dependencies. For example, after a shared library successfully builds, it can trigger the test pipelines of all applications that depend on it.

WHEN NOT TO USE IT Avoid this for simple, linear workflows. If your pipeline is small and all jobs run on every commit, a single configuration file is simpler to understand and maintain. Overusing dynamic pipelines can make the overall workflow difficult to trace and debug, as logic is spread across multiple files and projects.

ONE CANONICAL EXAMPLE A monorepo contains three services: api, frontend, and payments. A parent pipeline runs a script that detects changes. If only files in the frontend/ directory are modified, it dynamically generates and triggers a child pipeline defined in frontend-ci.yml. This avoids running unnecessary tests for the api and payments services, saving significant time and resources. The parent pipeline doesn't wait for the child's result unless configured with strategy: depend.