Cloud Composer (Airflow) on GCP Explained

Once a platform has more than a handful of pipelines, something has to decide what runs when, in what order, and what happens after a failure. Apache Airflow is the most widely used open source tool for that job, and running it yourself means running a scheduler, workers, a web server and a database. Cloud Composer is Google Cloud's managed version: Airflow, with the infrastructure handled for you.

A note on the name before anything else. Google is renaming the service. Its documentation now calls it Managed Service for Apache Airflow, or Managed Airflow, and says that Cloud Composer is evolving to become it (Google Cloud: Managed Airflow overview). The API and many identifiers still say Composer, and so will most people you work with for some time.

What an environment is

You do not get "an Airflow". You create an environment, which is a complete, self-contained Airflow deployment (Google Cloud: environment architecture):

• A bucket in your project holding DAG files, plugins and logs. Deploying a DAG means putting a Python file in the bucket's dags folder; the service synchronizes it to Airflow.
• Schedulers, workers and triggerers that parse DAGs and run tasks. Workers scale with demand.
• The Airflow web server, with access controlled by IAM.
• The Airflow database, a Cloud SQL instance that Google runs in a separate, managed project.

In the current generation, most of this is hidden. You see the bucket and the Airflow interface, not the machines.

Three generations, two of them ending

The versioning page gives 15 September 2026 as the planned end of life for Legacy Gen 1 and for versions 2.0.x of Gen 2 (Google Cloud: versioning overview). Any tutorial that begins by creating a Composer 1 environment is describing something you can no longer rely on. New work belongs on Gen 3.

A DAG for Google Cloud

A DAG is a Python file that declares tasks and the order between them. The Airflow tutorial linked at the end covers the concepts; here is what a typical Google Cloud DAG looks like. It waits for a day's files to arrive, loads them into a staging table and then runs a SQL procedure that builds the reporting table.

import datetime

from airflow import models
from airflow.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperator
from airflow.providers.google.cloud.sensors.gcs import GCSObjectExistenceSensor
from airflow.providers.google.cloud.transfers.gcs_to_bigquery import GCSToBigQueryOperator

BUCKET = "example-landing"

with models.DAG(
    "daily_trips",
    schedule="10 2 * * *",
    start_date=datetime.datetime(2026, 10, 1),
    catchup=False,
    default_args={"retries": 2, "retry_delay": datetime.timedelta(minutes=5)},
) as dag:
    wait_for_file = GCSObjectExistenceSensor(
        task_id="wait_for_file",
        bucket=BUCKET,
        object="trips/dt={{ ds }}/_SUCCESS",
        deferrable=True,
    )
    load_staging = GCSToBigQueryOperator(
        task_id="load_staging",
        bucket=BUCKET,
        source_objects=["trips/dt={{ ds }}/*.parquet"],
        destination_project_dataset_table="analytics-prod.staging.trips",
        source_format="PARQUET",
        write_disposition="WRITE_TRUNCATE",
    )
    build_report = BigQueryInsertJobOperator(
        task_id="build_report",
        configuration={"query": {"query": "CALL reporting.build_daily_sales('{{ ds }}')",
                                 "useLegacySql": False}},
        location="US",
    )
    wait_for_file >> load_staging >> build_report

The operators and their parameters are the documented ones (Google Cloud: write Airflow DAGs; Apache Airflow: Cloud Storage to BigQuery; Apache Airflow: Cloud Storage operators). The file was checked only as valid Python, not loaded by Airflow.

Four things in it carry lessons:

• The run date is a template. {{ ds }} is filled in by Airflow with the date the run is for, so a rerun or a backfill of an old date processes that date's files.
• The sensor is deferrable. While it waits for the file it does not occupy a worker.
• The load overwrites its staging table, so running it twice is harmless.
• The heavy work happens in BigQuery, not in Airflow. Each task is a request to another service.

That last point is the rule for using Airflow well on any platform. It is an orchestrator. Google's guide says to avoid CPU-heavy and memory-heavy tasks on the environment's workers. A DAG that reads a large file into pandas inside a task is using the scheduler as a compute engine, and it will starve everything else in the environment.

Deploying is a file copy (Google Cloud: add and update DAGs):

gcloud composer environments storage dags import \
  --environment analytics-airflow \
  --location asia-south1 \
  --source daily_trips.py

What it costs

An environment is not serverless. It runs, and bills, all day, whether or not any DAG is executing. The current generation is billed in DCU-hours, a unit covering the compute and memory the environment uses, at a list rate of USD 0.06 per DCU-hour in us-central1, plus a small charge for database storage (Google Cloud: Managed Airflow pricing).

The script works through Google's own pricing example and extends it to a month.

# What a Managed Airflow (Gen 3) environment costs, from the list rate on Google's pricing page for
# us-central1 on 2 October 2026 and the DCU figures in Google's own worked example. A calculation,
# not a bill: real usage depends on environment size, autoscaling and region.
RATE = 0.06                      # US dollars per DCU-hour

# 1. Google's example: 12 DCUs for 5 days, then 15 DCUs for 5 days.
dcu_hours = 12 * 5 * 24 + 15 * 5 * 24
print(f"Google's example: {dcu_hours:,} DCU-hours -> USD {dcu_hours * RATE:,.2f}")

# 2. The same environment left running for a whole month.
HOURS = 730
print(f"\n{'DCUs in use':>11} {'DCU-hours a month':>18} {'USD a month':>12} {'USD a day':>10}")
for dcus in (12, 15):
    month = dcus * HOURS
    print(f"{dcus:11} {month:18,} {month * RATE:12,.2f} {dcus * 24 * RATE:10,.2f}")

# 3. The cost is the same whether the environment runs one pipeline or fifty.
monthly = 12 * HOURS * RATE
print(f"\n{'daily pipelines':>15} {'USD per pipeline run':>21}")
for pipelines in (1, 10, 50):
    print(f"{pipelines:15} {monthly / (pipelines * 30):21.2f}")

Google's example: 3,240 DCU-hours -> USD 194.40

DCUs in use  DCU-hours a month  USD a month  USD a day
         12              8,760       525.60      17.28
         15             10,950       657.00      21.60

daily pipelines  USD per pipeline run
              1                 17.52
             10                  1.75
             50                  0.35

Google's example of ten days comes to USD 194.40, and the script reproduces it. The same environment kept for a month costs about USD 526 at 12 DCUs. Those DCU figures are Google's illustration, not a quote for your environment, but the shape of the result holds: the cost is a fixed monthly sum, and it does not depend on how many pipelines use it.

The third table is the one that should drive the decision. With one daily pipeline, each run effectively costs over USD 17 in orchestration alone. With fifty, it costs 35 cents.

When it is worth it

Managed Airflow earns its cost when you have:

• Many pipelines with dependencies between them, across several services.
• A need for backfills, retries with alerts and a visible history of every run.
• Sensors and branching: wait for a file, run this only if that succeeded.
• A team that already writes Airflow DAGs, or wants pipelines as reviewed Python code.

It is too much when you have:

• A few independent SQL statements. Use scheduled queries in BigQuery, which cost nothing to orchestrate.
• SQL transformations with dependencies, all inside BigQuery. A SQL transformation tool handles that without an always-on environment. See Dataform vs dbt.
• One event-driven job. A function triggered by the event is simpler.

Running one well

• Keep work out of the workers. Call BigQuery, Dataflow or the Spark service, and let them do the processing.
• Prefer deferrable operators for anything that waits.
• Make every task rerunnable. Overwrite a partition or merge; never blindly append.
• Give the environment its own service account with only the roles its DAGs need. See service accounts and workload identity.
• Version the DAGs in Git and deploy from a pipeline, not by hand.
• Size down. Autoscaling has a lower limit; set it to what you need, and delete environments used for experiments.

For Airflow itself, DAGs, tasks, scheduling and testing, see the Airflow tutorial.

Orchestration with Airflow is covered in the Data Engineering course.

Sources

• Managed Airflow overview (Google Cloud Documentation) — Google Cloud
• Managed Service for Apache Airflow versioning overview (Google Cloud Documentation) — Google Cloud
• Environment architecture (Google Cloud Documentation) — Google Cloud
• Write Airflow DAGs (Google Cloud Documentation) — Google Cloud
• Google Cloud Storage transfer operator to BigQuery (Apache Airflow) — Apache Airflow
• Google Cloud Storage operators (Apache Airflow) — Apache Airflow
• Managed Service for Apache Airflow pricing (Google Cloud) — Google Cloud
• Add and update DAGs (Google Cloud Documentation) — Google Cloud