AWS Glue Tutorial: Serverless ETL on AWS

AWS Glue is the service most AWS data pipelines pass through at some point. AWS describes it as a serverless data integration service for discovering, preparing, moving and integrating data from multiple sources (Glue docs: what is AWS Glue). In practice it is a small family of tools under one name, and the first hurdle is knowing which part does what.

This tutorial explains the parts, shows how they join into a pipeline, and defines one in code.

The parts of Glue

The concepts page names four key components (Glue docs: concepts):

Around these sit workflows, which group crawlers, jobs and triggers into one monitored unit; Glue Studio, the visual editor; and Data Quality, for rules checked against data.

One sentence from the documentation prevents a lot of confusion: tables and databases in the Data Catalog contain metadata; they do not contain data. Glue does not store your data. The data stays in S3 or a database, and Glue describes it and runs code against it.

How the parts make a pipeline

The typical flow, from the same page:

1. Catalogue the source. A crawler scans raw files in S3 and creates table definitions. Or you define the tables yourself.
2. Write a job. A script reads the source table, transforms the data and writes to a target, usually Parquet in S3.
3. Run it. On demand, or from a trigger.
4. Catalogue the output, so it can be queried.
5. Query it. Athena, Redshift and EMR read the same catalog.
6. Monitor runs and logs.

Each step has its own article here: the Data Catalog and crawlers, writing Glue jobs in PySpark, and Glue Studio's visual editor.

A pipeline, defined in code

Clicking through the console is fine for learning. A pipeline you intend to keep should be defined in code, so it can be reviewed and recreated. The following uses boto3 to create a database, a crawler over a raw prefix, a Spark job and a nightly trigger. It runs against moto, a local emulator of the Glue API: the definitions are recorded and read back, but nothing is crawled and no Spark job runs.

import os
import boto3
from moto import mock_aws

os.environ.update(AWS_ACCESS_KEY_ID="testing", AWS_SECRET_ACCESS_KEY="testing",
                  AWS_DEFAULT_REGION="us-east-1")
ROLE = "arn:aws:iam::123456789012:role/GlueJobRole"


@mock_aws
def main():
    glue = boto3.client("glue")

    # 1. A database to hold table definitions
    glue.create_database(DatabaseInput={"Name": "lake"})

    # 2. A crawler that will catalogue the raw files
    glue.create_crawler(
        Name="raw-trips-crawler", Role=ROLE, DatabaseName="lake",
        Targets={"S3Targets": [{"Path": "s3://amzn-s3-demo-bucket/raw/tlc/yellow/"}]},
        SchemaChangePolicy={"UpdateBehavior": "LOG", "DeleteBehavior": "LOG"})

    # 3. A Spark job: the script lives in S3, compute is declared here
    glue.create_job(
        Name="trips-to-curated", Role=ROLE,
        Command={"Name": "glueetl", "ScriptLocation": "s3://amzn-s3-demo-bucket/scripts/trips_to_curated.py",
                 "PythonVersion": "3"},
        GlueVersion="5.1", WorkerType="G.1X", NumberOfWorkers=2,
        DefaultArguments={"--job-bookmark-option": "job-bookmark-enable", "--run_month": "2024-01"},
        Timeout=30, MaxRetries=1)

    # 4. A schedule: 02:00 UTC every day
    glue.create_trigger(
        Name="nightly-trips", Type="SCHEDULED", Schedule="cron(0 2 * * ? *)",
        Actions=[{"JobName": "trips-to-curated"}], StartOnCreation=True)

    job = glue.get_job(JobName="trips-to-curated")["Job"]
    print("job      :", job["Name"], "| type", job["Command"]["Name"], "| Glue", job["GlueVersion"],
          "|", job["NumberOfWorkers"], "x", job["WorkerType"], "| timeout", job["Timeout"], "min")
    print("arguments:", job["DefaultArguments"])
    crawler = glue.get_crawler(Name="raw-trips-crawler")["Crawler"]
    print("crawler  :", crawler["Name"], "->", crawler["Targets"]["S3Targets"][0]["Path"],
          "| schema changes:", crawler["SchemaChangePolicy"]["UpdateBehavior"])
    trigger = glue.get_trigger(Name="nightly-trips")["Trigger"]
    print("trigger  :", trigger["Name"], trigger["Type"], trigger["Schedule"], "->",
          [a["JobName"] for a in trigger["Actions"]])


main()

job      : trips-to-curated | type glueetl | Glue 5.1 | 2 x G.1X | timeout 30 min
arguments: {'--job-bookmark-option': 'job-bookmark-enable', '--run_month': '2024-01'}
crawler  : raw-trips-crawler -> s3://amzn-s3-demo-bucket/raw/tlc/yellow/ | schema changes: LOG
trigger  : nightly-trips SCHEDULED cron(0 2 * * ? *) -> ['trips-to-curated']

What the definitions say:

• The crawler targets one S3 prefix and is told to log schema changes instead of applying them, so a surprise in the source cannot silently alter the table.
• The job is of type glueetl, a Spark job. Its script sits in S3. It runs on Glue 5.1 with two G.1X workers, has a 30-minute timeout and one retry.
• Default arguments switch on job bookmarks, so each run processes only new data, and pass a parameter the script can read.
• The trigger is scheduled with a cron expression. Glue schedules are in UTC, with six fields and a minimum precision of five minutes (Glue docs: time-based schedules).

In a real project the same definitions are usually written in CloudFormation, Terraform or the CDK. The properties are the same; boto3 shows them without another tool to learn.

Starting things: triggers and workflows

A trigger fires in one of three ways (Glue docs: triggers):

• Scheduled, on a cron expression.
• Conditional, when watched jobs or crawlers finish with a given status, such as succeeded or failed. This is how you say "run the load only after the cleaning job succeeds".
• On demand, when you start it.

For complex multi-job operations, the documentation prefers workflows. A workflow groups crawlers, jobs and triggers, records the progress of each, and draws the run as a graph. It can start on a schedule, on demand, or from an Amazon EventBridge event such as a new object arriving in S3 (Glue docs: workflows). When a pipeline has to coordinate services outside Glue, a general orchestrator is the better fit; Glue workflows vs Step Functions compares them.

Job types in brief

Choosing a Spark job for a task that moves a few megabytes is a common waste; the Python shell type or a Lambda function costs far less.

What you pay for

Glue has no fixed fee. The pricing page describes three meters (AWS: Glue pricing):

• Jobs: an hourly rate per DPU, billed by the second, with a one-minute minimum on current versions. A DPU is 4 vCPUs and 16 GB of memory. Spark jobs need at least 2 DPUs and are given 10 by default.
• Crawlers: an hourly rate, billed by the second, for the time they run.
• Data Catalog: a monthly charge for objects stored and requests, with the first million objects and first million accesses free.

The lever you control is DPUs multiplied by run time. Set the worker count on purpose, give every job a timeout, and do not crawl on a schedule when the data has not changed.

Settings to get right on day one

• A dedicated IAM role per job, limited to the buckets and tables it needs.
• Worker count and timeout, not the defaults.
• Job bookmarks enabled for incremental sources, or an explicit date parameter.
• Parquet output, partitioned, with partitions overwritten per run.
• Crawler include paths aimed at one dataset each.
• Logs and metrics switched on, and looked at.

Where Glue fits, and where it does not

Glue suits scheduled batch ETL into a data lake or warehouse, especially for teams that do not want to run clusters. It is less suited to very small tasks, where a Lambda function is simpler, and to workloads that need full control of the Spark environment or frameworks beyond Spark, where EMR fits. Glue vs EMR vs Lambda for ETL sets out the choice, and the AWS roadmap for data engineers places Glue among the other services.

AWS Glue, Amazon S3 and ETL pipeline design are covered in the Data Engineering course.

Sources

• What is AWS Glue? (AWS Glue User Guide) — AWS Documentation
• AWS Glue concepts (AWS Glue User Guide) — AWS Documentation
• AWS Glue triggers (AWS Glue User Guide) — AWS Documentation
• Overview of workflows in AWS Glue (AWS Glue User Guide) — AWS Documentation
• Time-based schedules for jobs and crawlers (AWS Glue User Guide) — AWS Documentation
• AWS Glue pricing (Amazon Web Services) — Amazon Web Services