Argo Workflows: Run Complex Jobs on Kubernetes
Think of Argo Workflows as a script runner for Kubernetes, where each command is a container. It runs multi-step jobs like CI/CD or ML pipelines. The footgun is treating it like a full CI server; it's just an engine and lacks features like Git polling.
WHY IT EXISTS: Kubernetes excels at running long-lived services, but it lacks a native concept for running a sequence of tasks that eventually complete, like a batch job or a data processing pipeline. Argo Workflows fills this gap by providing a Kubernetes-native engine to define, schedule, and manage these finite, often complex, workflows.
THE MENTAL MODEL: Treat Argo Workflows as a way to define a script or a Makefile directly in Kubernetes YAML. Instead of shell commands, each step is a container. You define a Directed Acyclic Graph (DAG) of these steps, specifying dependencies, inputs, and outputs. The entire workflow is a Kubernetes object, manageable with tools like kubectl.
HOW IT WORKS: Argo Workflows introduces Custom Resource Definitions (CRDs) to your cluster, primarily the Workflow CRD. You define a workflow by creating a YAML manifest for a Workflow resource. This manifest specifies a series of templates (container specifications) and an entrypoint that defines the execution graph. The Argo controller watches for these resources and orchestrates the creation and deletion of pods to execute each step.
WHEN TO USE IT: Use Argo Workflows for any task that can be modeled as a DAG of containers. This is common in MLOps for multi-stage training pipelines, in data engineering for ETL jobs, for complex CI/CD logic, and for general-purpose infrastructure automation.
WHEN NOT TO USE IT: Avoid Argo for simple, single-container jobs where a standard Kubernetes Job or CronJob would suffice. It is not a drop-in replacement for a full CI/CD platform like Jenkins or GitLab CI if you need features like SCM integration, webhooks, and user management out of the box. Argo provides the engine; you build the surrounding product.
ONE CANONICAL EXAMPLE: A classic machine learning pipeline. Step 1: A container downloads and preprocesses a dataset. Step 2: A container loads the data, trains a model, and saves the model file. Step 3: A container takes the model and runs it against a test set, outputting metrics. Argo ensures these steps run in order, passing artifacts between them.
Read the original → argoproj.github.io
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.