DAG
The definition of a pipeline's tasks and their dependencies — which steps must finish before others start, with no cycles.
Lesson 3 of 3 · ⭐ Gap priority
Data Pipelines & Transformation
A nightly pipeline silently stopped backfilling three days ago, and nobody noticed until someone asked why a dashboard looked stale.
Orchestration vs. transformation
Airflow and dbt solve different problems, and are almost always used together, not as alternatives.
Airflow orchestrates — it decides when tasks run and in what order, and retries or alerts when they fail. dbt defines what a specific transformation actually does once it runs. A typical nightly job is an Airflow task that triggers a dbt run, not one tool replacing the other.
Core Airflow concepts
Four terms cover most of what you'll need to read someone else's pipeline.
Task
A single unit of work in the DAG — run a script, execute a dbt model, load a file.
Schedule / trigger
When the DAG runs — a cron-like schedule, or triggered by an external event such as a file landing or an API call.
Backfill
Re-running a DAG for past dates to fill in or fix historical data, instead of only ever running forward from now.
Exercises
Look at the sanitized DAG run history below, then work through both exercises.
DAG: nightly_customer_sync
2026-07-19 task: extract_customers SUCCESS
2026-07-19 task: load_to_warehouse FAILED (upstream schema changed: dropped column "region")
2026-07-20 task: extract_customers SUCCESS
2026-07-20 task: load_to_warehouse FAILED (same error, no alert configured on this DAG)
2026-07-21 task: extract_customers SUCCESS
2026-07-21 task: load_to_warehouse FAILED (same error)Concept check
Answer in your head first, then click a question to reveal the answer.
Scenario checklist
Work through the sample DAG run history and check each item off. Progress is saved in your browser.
Go deeper
These primers are a starting point. Read the primary sources to build real fluency.
Apache Airflow Docs: DAGsThe core scheduling/dependency concept this lesson is built aroundApache Airflow Docs: Core conceptsTasks, operators, and DAG runs, from the official documentationdbt Developer Hub: About dbt modelsWhat the transformation step (as opposed to orchestration) actually looks like
Training Notes
What building this project taught me.
This is a flagged gap-priority module
This lesson continues the same gap as Modern Data Stack Foundations — modern data stack fluency, Tom's #4 self-identified gap — but on the orchestration and pipeline side rather than the warehouse side.
The failure pattern below is common, not exaggerated
A pipeline that fails silently for days because no alert was configured is one of the most frequently reported real-world data engineering incidents — worth treating as a default risk to guard against, not an edge case.