AWS Glue Jobs: PySpark on Serverless Spark

An AWS Glue job is a Spark program that AWS runs for you. You supply a script and a few settings; Glue starts a Spark environment, runs the script, and shuts everything down. There is no cluster to create or patch, and billing stops when the job does. If you know PySpark, you already know most of what a Glue job contains. The rest is a thin layer of Glue-specific setup, and a few decisions about how the job tracks its work and how much compute it gets.

For where jobs sit among Glue's other parts, see the AWS Glue tutorial. This article is about the script.

What defines a job

Beyond the script, a job has properties (Glue docs: job properties):

• Type. Spark for batch ETL, Spark Streaming, or Python shell for small non-Spark scripts.
• IAM role. The permissions the job runs with: which buckets and catalog tables it may read and write.
• Glue version. This fixes the Spark and Python versions. Glue 5.1, the default for new jobs, provides Spark 3.5.6 and Python 3.11; Glue 6.0 provides Spark 4.1.1 and Python 3.13.
• Worker type and number of workers. How much compute the job gets.

Structure the script in three parts

A maintainable Glue script separates what is Glue from what is Spark:

1. Setup: read arguments, create the contexts, initialise the job.
2. Transform: a function that takes DataFrames and returns a DataFrame, with no Glue classes in it.
3. Write and commit.

The reason is testing. The awsglue library exists only inside the Glue runtime, so anything that imports it cannot run on your machine or in CI. A transform written as plain PySpark can.

The transform, run locally

Here is a transform that computes daily trips and fares per pickup borough from New York taxi data. It is run with open-source PySpark in local mode, not on Glue.

from pyspark.sql import DataFrame, SparkSession, functions as F


def transform(trips: DataFrame, zones: DataFrame) -> DataFrame:
    """Daily trips and fares per pickup borough. Pure Spark: no Glue classes, so it runs anywhere."""
    clean = (trips
             .where((F.col("tpep_pickup_datetime") >= "2024-01-01") & (F.col("tpep_pickup_datetime") < "2024-02-01"))
             .where(F.col("fare_amount") >= 0)
             .withColumn("pickup_date", F.to_date("tpep_pickup_datetime")))
    return (clean.join(F.broadcast(zones), clean.PULocationID == zones.LocationID, "left")
            .groupBy("pickup_date", "Borough")
            .agg(F.count("*").alias("trips"), F.round(F.sum("fare_amount"), 2).alias("fares")))


if __name__ == "__main__":
    spark = (SparkSession.builder.master("local[4]").appName("daily_borough")
             .config("spark.sql.shuffle.partitions", "4").config("spark.ui.enabled", "false").getOrCreate())
    spark.sparkContext.setLogLevel("ERROR")
    trips = spark.read.parquet("yellow_tripdata_2024-01.parquet")
    zones = spark.read.option("header", True).csv("taxi_zone_lookup.csv")

    result = transform(trips, zones).cache()
    print("rows:", result.count(), "| days:", result.select("pickup_date").distinct().count())
    result.where("pickup_date = '2024-01-15'").orderBy(F.desc("trips")).show(truncate=False)
    spark.stop()

rows: 245 | days: 31
+-----------+-------------+-----+---------+
|pickup_date|Borough      |trips|fares    |
+-----------+-------------+-----+---------+
|2024-01-15 |Manhattan    |65082|943665.94|
|2024-01-15 |Queens       |10084|532019.08|
|2024-01-15 |Brooklyn     |475  |14255.13 |
|2024-01-15 |Unknown      |301  |7269.08  |
|2024-01-15 |Bronx        |94   |3182.46  |
|2024-01-15 |N/A          |50   |4338.93  |
|2024-01-15 |EWR          |16   |1672.05  |
|2024-01-15 |Staten Island|1    |75.0     |
+-----------+-------------+-----+---------+

The function filters out-of-month and negative-fare rows, joins the small zone table with a broadcast hint, and aggregates: 245 rows for 31 days. On 15 January, Manhattan accounts for 65,082 trips. The values can be checked, a unit test can assert them, and nothing here needed an AWS account. The local run used Spark 3.5, the same line as Glue 5.1; match versions when you test, because behaviour differs between Spark 3 and 4. The PySpark tutorial covers the DataFrame operations used.

The Glue wrapper

The same function inside a Glue job. This is written from the documentation and has not been run; it only runs in the Glue runtime:

import sys
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext

from transforms import transform          # the function shown above, shipped with the job

args = getResolvedOptions(sys.argv, ["JOB_NAME"])
glue_context = GlueContext(SparkContext.getOrCreate())
job = Job(glue_context)
job.init(args["JOB_NAME"], args)

trips = glue_context.create_dynamic_frame.from_catalog(
    database="lake", table_name="trips_raw", transformation_ctx="trips_raw").toDF()
zones = glue_context.create_dynamic_frame.from_catalog(
    database="lake", table_name="taxi_zones").toDF()

result = transform(trips, zones)

(result.write.mode("overwrite")
       .partitionBy("pickup_date")
       .parquet("s3://amzn-s3-demo-bucket/curated/daily_borough/"))

job.commit()

Sources are read from Data Catalog tables by name, so the script holds no S3 paths or schemas for its inputs; the Glue Data Catalog explains those tables.

DynamicFrame or DataFrame?

Reading from the catalog returns a DynamicFrame, Glue's own structure. AWS's explanation of why it exists: a DataFrame needs a schema before data is loaded, and schema inference struggles with messy data in which the same field has different types in different records. In a DynamicFrame each record is self-describing, so no schema is required up front, and conflicts can be resolved explicitly (Glue docs: DynamicFrame).

A common pattern is the one above: read as a DynamicFrame to use the catalog and bookmarks, call toDF(), do the real work in DataFrames, and write. fromDF converts back when a Glue writer is needed.

Job bookmarks: not reprocessing old data

A scheduled job should not reread everything each time. Job bookmarks are Glue's answer: the job persists state from each run and uses it to skip data it has already processed (Glue docs: job bookmarks).

Three requirements, all easy to miss:

• Enable them. Bookmarks are disabled by default; the job then processes the entire dataset on every run.
• Initialise and commit. Call job.init at the start and job.commit at the end; the commit is what saves the state.
• Name each source. Pass a transformation_ctx to each source; it is the key under which that source's state is stored; without it, bookmarks are not enabled for the source.

Bookmarks track what was read, not what was written. If a run fails after writing some output, a rerun can duplicate it unless the write is itself safe to repeat. Overwriting whole partitions, as the script does, is the simple protection; idempotency in data pipelines explains why that matters.

Workers, DPUs and the bill

Compute is measured in DPUs: one DPU is 4 vCPUs and 16 GB of memory. The standard worker types:

Jobs are charged per DPU-hour, billed by the second, with a one-minute minimum on current versions and no charge for startup time. A Spark job needs at least 2 DPUs and is allocated 10 by default (AWS: Glue pricing). That default is worth noticing: a small job left on default capacity pays for ten DPUs it may not use.

Two controls help. Set the number of workers deliberately, starting small. Or enable Auto Scaling, with which Glue adds and removes workers during the run, up to your maximum, according to the parallelism of each stage (Glue docs: auto scaling).

Practices that pay off

• Keep transforms free of Glue imports and test them locally.
• Pass run parameters as job arguments, such as the date to process, instead of using the clock.
• Write Parquet, partitioned, and overwrite by partition.
• Filter early and select only needed columns, so less data is shuffled.
• Watch for small files. Many output partitions times many Spark partitions makes thousands of tiny objects; repartition before writing.
• Read the run metrics after the first runs, and resize workers from evidence.

Glue is one of three common places to run ETL code on AWS; Glue vs EMR vs Lambda compares them.

Apache Spark, PySpark and AWS Glue are covered in the Data Engineering course.

Sources

• Configuring job properties for Spark jobs in AWS Glue (AWS Glue User Guide) — AWS Documentation
• DynamicFrame class (AWS Glue User Guide) — AWS Documentation
• Tracking processed data using job bookmarks (AWS Glue User Guide) — AWS Documentation
• Using job bookmarks (AWS Glue User Guide) — AWS Documentation
• AWS Glue pricing (Amazon Web Services) — Amazon Web Services
• Using auto scaling for AWS Glue (AWS Glue User Guide) — AWS Documentation