AWS Glue vs EMR vs Lambda for ETL

On AWS there are three common places to put transformation code: an AWS Lambda function, an AWS Glue job, or a cluster or application in Amazon EMR. Tutorials tend to pick one and not say why. The choice is not a matter of taste. Each service has limits and a billing model that make it right for one size and shape of work and wrong for another.

The three in one paragraph each

Lambda runs a function in response to an event. It starts in moments, needs no setup, and stops existing when it returns. It is plain Python, or another runtime, on a single small machine. See AWS Lambda explained.

Glue runs Spark jobs, or small Python scripts, on capacity that AWS provisions per run. It adds a Data Catalog, crawlers, job bookmarks and a visual editor. There is no cluster to manage. See the AWS Glue tutorial.

EMR runs the open-source big data frameworks, Spark, Hive, Presto, HBase and others, on clusters you configure, on Kubernetes, or in a serverless mode. It gives the most control and asks the most of you. See Amazon EMR explained.

The limits that decide most cases

The Lambda figures come from its quotas page (Lambda docs: quotas); the Glue figures from its job properties and pricing pages (Glue docs: job properties).

Read the first two rows together. Lambda cannot hold a dataset larger than its memory and cannot work for more than fifteen minutes. That rules it out for anything a data engineer would call big. Glue and EMR distribute work across machines, so their limit is what you are willing to pay for.

Decide by data size first

Kilobytes to a few hundred megabytes per run. Lambda. A file arrives, a function validates and converts it. A Spark job here spends most of its time starting up, and Glue's Spark jobs need at least 2 DPUs with a one-minute minimum (AWS: Glue pricing). A Glue Python shell job, which runs without Spark on 0.0625 or 1 DPU, covers the same ground for scheduled work that does not fit Lambda's 15 minutes (Glue docs: Python shell jobs).

Gigabytes to hundreds of gigabytes per run. Glue. This is distributed work, and Glue runs Spark without asking you to design a cluster.

Terabytes, or sustained heavy processing. Glue still works; EMR starts to matter, because at this scale control over instance types, Spot capacity and Spark configuration turns into real money and real speed.

Size is a guide, not a rule. A 10 GB job that runs once a day is easy on Glue. The same job running every five minutes is a different question.

Then by control and skills

Choose Glue when:

• You want scheduled Spark ETL with the least to operate.
• The Data Catalog, crawlers and job bookmarks are useful to you.
• The team is small or new to Spark; the visual editor and SQL transforms lower the barrier.
• Workloads are intermittent: capacity exists only while a job runs.

Choose EMR when:

• You need a specific Spark version, custom configuration, or libraries and native dependencies installed on the nodes.
• You use frameworks other than Spark: Hive, Presto, HBase, Flink.
• Jobs are large and long enough that instance choice and Spot Instances cut the bill substantially (AWS: EMR pricing).
• You are moving existing Hadoop or Spark workloads to AWS and want them to run as they are.
• You need a long-lived cluster for interactive analysis.

Choose Lambda when:

• The work is event-driven and small per event.
• You need to react within seconds.
• The task is glue between services: check, route, start, notify.

EMR Serverless narrows the gap between the first two. It runs Spark applications without clusters to configure, obtaining and releasing resources per job (EMR docs: EMR Serverless), while keeping EMR's open-source runtime. If your reason for choosing Glue was "no clusters" and your reason for hesitating was "I want plain Spark", it is worth evaluating.

Cost: three different meters

• Lambda charges for requests and for run time multiplied by memory. Idle costs nothing. Very cheap for sporadic small work; less so for constant high-volume processing.
• Glue charges per DPU-hour, by the second. Simple to reason about: workers multiplied by run time. The default of 10 DPUs is the usual source of waste.
• EMR on EC2 charges an EMR rate on top of the EC2 instances, per second. More to manage, and more ways to save: right-sized instances, Spot for task nodes, clusters that terminate after the job.

No service is cheapest in general. A rough rule that holds often: Lambda for small and occasional, Glue for medium and scheduled, EMR for large and frequent. Estimate with your own run times, because rates vary by Region and change.

The pattern that uses all three

Mature pipelines rarely pick one:

1. A file lands in S3. A Lambda function checks its name, size and header, and starts the pipeline.
2. A Glue job cleans the raw data and writes partitioned Parquet.
3. A nightly EMR job, or a larger Glue job, builds the heavy aggregates over months of history.
4. An orchestrator such as Step Functions ties them together and handles failure.

Each engine does what it is sized for. Building an ETL pipeline on AWS works through such a pipeline, and AWS Step Functions covers the orchestration.

Decision table

Mistakes to avoid

• Forcing big data through Lambda by chunking it into thousands of invocations, when a Spark job would be simpler.
• Spark for tiny data. A ten-DPU job for a 5 MB file.
• A permanent EMR cluster for a job that runs twenty minutes a day.
• Choosing by familiarity alone. The right engine for the last project is not automatically right for this one.
• Locking logic to the engine. Keep transformations as plain PySpark or Python functions, so moving from Glue to EMR, or from Lambda to a container, is a small change.

ETL and ELT pipelines with Apache Spark and AWS services including Glue and EMR are covered in the Data Engineering course.

Sources

• Lambda quotas (AWS Lambda Developer Guide) — AWS Documentation
• Configuring job properties for Spark jobs in AWS Glue (AWS Glue User Guide) — AWS Documentation
• Configuring job properties for Python shell jobs in AWS Glue (AWS Glue User Guide) — AWS Documentation
• AWS Glue pricing (Amazon Web Services) — Amazon Web Services
• What is Amazon EMR? (Amazon EMR Management Guide) — AWS Documentation
• What is Amazon EMR Serverless? (Amazon EMR Serverless User Guide) — AWS Documentation
• Amazon EMR pricing (Amazon Web Services) — Amazon Web Services