Apache Airflow: Code-Defined Data Pipelines
Airflow lets you define, schedule, and monitor complex data workflows as code, replacing brittle cron jobs. It's used for ETL jobs or ML training pipelines. The footgun is treating it as a data processing engine; it's an orchestrator, not the worker.
Why it exists
As data operations grow, managing them with simple scripts and cron jobs becomes fragile and hard to observe. Companies like Airbnb faced this challenge with their complex workflows, needing a systematic way to define, schedule, monitor, and retry tasks. Airflow was created to solve this problem of managing complexity in data pipelines.
The mental model
Airflow is an orchestrator, not a data processing engine. Think of it as a conductor for a data orchestra. The conductor doesn't play any instruments but directs the various sections—a Spark job, a database query, an API call—telling them when to start, what to do, and how to recover from errors. Your workflow is the score, written in Python.
How it works
You define workflows as Directed Acyclic Graphs (DAGs) using Python scripts. A DAG is a collection of tasks with defined dependencies, ensuring they run in the correct order. For example, a task to transform data only runs after the task to extract it has succeeded. Airflow's scheduler executes these tasks on a schedule, while the web UI provides a powerful interface to monitor progress, inspect logs, and manage the state of your pipelines.
When to use it
Airflow excels at managing batch-oriented workflows with complex dependencies that run on a schedule. It is the go-to tool for building robust ETL/ELT pipelines, automating machine learning model training and deployment, and generating complex reports. Its strengths are its ability to handle retries, alerting, and backfilling historical data runs.
When not to use it
Airflow is not built for streaming or real-time data processing. For use cases requiring millisecond latency, tools like Apache Flink or Kafka Streams are more appropriate. A major footgun is running heavy data processing directly on Airflow workers. Always offload intensive computation to specialized systems like Spark, Snowflake, or BigQuery, and use Airflow only to trigger and monitor that work.
One canonical example
A daily e-commerce analytics pipeline. The DAG first defines a task to extract sales and user activity data from production databases. Once complete, it triggers parallel tasks: one to process sales in a Spark cluster and another to analyze user behavior. When both are finished, a final task joins the results and loads them into a data warehouse for analysts. Airflow manages this entire dependency chain, retries any failed step, and sends an alert if the final data isn't ready by 9 AM.
Interview question
Which statement best describes Apache Airflow's primary function in a data engineering ecosystem?
- a.Orchestrating, scheduling, and monitoring complex data workflows across distributed systems.Correct
- b.Executing intensive data transformations and computations on its worker nodes.
- c.Providing a real-time data streaming and processing platform for low-latency analytics.
- d.Serving as a simple cron job replacement for running isolated, single-step scripts.
Why? this is the answer
Airflow is described as an orchestrator, not a data processing engine. Its role is to direct tasks across various systems, schedule them, and monitor their progress. Running heavy data processing directly on Airflow workers is explicitly identified as a 'major footgun'.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on data engineering — each one lists the topics its interview covers.
See open roles