Orchestration: Airflow vs Dagster vs Prefect

Airflow, Dagster and Prefect all schedule work, run steps in order, retry failures and show you what happened. They differ in the mental model they ask you to adopt: Airflow thinks in tasks, Prefect in Python functions, Dagster in data assets. The differences are much clearer in code than in feature lists, so this comparison writes one small pipeline three ways.

The pipeline

Take one day of New York taxi trips (NYC TLC: trip record data), drop bad fares, and summarise by pickup zone. The Airflow version is built and run in the Airflow tutorial, where it produces 216 zones and 76,103 trips for 15 January 2024. Here is the same logic in Prefect and in Dagster, in one script:

import os
import urllib.request
from pathlib import Path

os.environ.setdefault("PREFECT_LOGGING_LEVEL", "ERROR")       # keep the output to this script's own prints
os.environ.setdefault("PREFECT_SERVER_ANALYTICS_ENABLED", "false")

import dagster as dg
import pandas as pd
from prefect import flow, task

SOURCE = "yellow_tripdata_2024-01.parquet"
if not Path(SOURCE).exists():
    urllib.request.urlretrieve(f"https://d37ci6vzurychx.cloudfront.net/trip-data/{SOURCE}", SOURCE)

# the business logic is plain Python, shared by both orchestrators
def read_day(day):
    trips = pd.read_parquet(SOURCE, columns=["tpep_pickup_datetime", "PULocationID", "fare_amount"])
    return trips[trips["tpep_pickup_datetime"].dt.strftime("%Y-%m-%d") == day]

def summarise(trips):
    clean = trips[trips["fare_amount"] >= 0]
    return clean.groupby("PULocationID").agg(trips=("fare_amount", "size"), fares=("fare_amount", "sum")).reset_index()

# Prefect: flows and tasks. You describe the work, and call it like normal Python.
@task(retries=2)
def extract(day):
    return read_day(day)

@task
def transform(trips):
    return summarise(trips)

@flow
def taxi_flow(day):
    return transform(extract(day))

zones = taxi_flow("2024-01-15")
print(f"Prefect flow:  {len(zones)} zones, {int(zones['trips'].sum())} trips")

# Dagster: assets. You describe the data that should exist, and Dagster works out the order.
@dg.asset
def day_trips():
    return read_day("2024-01-15")

@dg.asset
def zone_summary(day_trips):                       # depends on day_trips by naming it as a parameter
    return summarise(day_trips)

result = dg.materialize([day_trips, zone_summary])
summary = result.output_for_node("zone_summary")
print(f"Dagster assets: {len(summary)} zones, {int(summary['trips'].sum())} trips | success: {result.success}")
print("assets materialised:", sorted(e.asset_key.to_user_string() for e in result.get_asset_materialization_events()))

Prefect flow:  216 zones, 76103 trips
Dagster assets: 216 zones, 76103 trips | success: True
assets materialised: ['day_trips', 'zone_summary']

Both return the same result as the Airflow DAG, and neither needed any service set up beforehand: each ran from this one Python script on a laptop. The business logic, two plain functions, is identical; only the wrapping differs, and the wrapping is the point.

Airflow: a scheduled graph of tasks

Airflow represents a workflow as a DAG of tasks, run by a scheduler and tracked in a metadata database (Airflow docs: architecture). You declare tasks and their order; Airflow decides when each runs and records every attempt. Its strengths come from age and adoption: a large library of ready-made operators for almost every system, managed services on each major cloud, and a deep pool of people who know it. Its costs are the moving parts, several services and a database, and a model centred on tasks, which says nothing about the data each task produces unless you add that yourself.

Prefect: Python that happens to be orchestrated

Prefect describes itself as an orchestration engine that turns Python functions into production-grade pipelines with minimal friction (Prefect docs: get started). In the script, taxi_flow("2024-01-15") is an ordinary function call: it runs immediately, takes arguments and returns a DataFrame. The decorators add state tracking and retries around it. Because the flow is real Python executed at run time, loops, branches and tasks created from data all work naturally, which suits dynamic and event-driven workflows. Scheduling and remote execution are added afterwards through deployments, on Prefect Cloud or a self-hosted server.

Dagster: declare the data, not the steps

Dagster's core idea is the asset: an object in persistent storage, such as a table, file or model, defined in code along with what it depends on (Dagster docs: assets). In the script, zone_summary declares its dependency simply by naming day_trips as a parameter; nothing says "run this after that". Dagster derives the order from the data dependencies, and the output lists the two assets it materialised. The pay-off is that lineage, freshness and data quality checks attach to the assets themselves, so the orchestrator's view of the world is the tables people actually use. The cost is a steeper first step: you must think in assets and learn Dagster's concepts before the first pipeline runs.

Side by side

How to choose

• Joining a team that already runs Airflow, or needing the widest set of integrations and managed options: Airflow. Familiarity and ecosystem outweigh elegance.
• A Python-heavy team that wants the shortest path from a script to a scheduled, retried, observable job, or whose workflows are dynamic and event-driven: Prefect.
• A team building a data platform around tables and models, especially with dbt, that wants lineage and data quality in the orchestrator: Dagster; the dbt tutorial covers the tool it most often orchestrates.
• A small project: any of the three will do. The cost of switching later is mostly the wrapping, as the example shows, provided the business logic stays in plain functions.

What matters more than the tool

Whichever you choose, the same properties make pipelines dependable: steps that can be rerun safely, runs driven by the period they are for and not the wall clock, checks that stop bad data, as in data quality testing, and alerts that reach a person. An orchestrator makes these easier to implement; none of them makes a non-idempotent task safe.

Apache Airflow, workflow scheduling, DAG creation and pipeline monitoring are part of the data pipelines and orchestration module of the Data Engineering course.

Sources

• Architecture overview (Apache Airflow documentation) — Apache Airflow
• Assets (Dagster documentation) — Dagster
• Get started (Prefect documentation) — Prefect
• TLC Trip Record Data (NYC Taxi and Limousine Commission) — NYC Taxi and Limousine Commission