Amazon EMR Explained: Big Data on AWS

Before serverless data services existed, running Spark or Hadoop meant building a cluster: provisioning machines, installing a matching set of frameworks, and keeping them healthy. Amazon EMR took that work over. AWS describes it as a managed cluster platform that simplifies running big data frameworks, such as Apache Hadoop and Apache Spark, on AWS (EMR docs: what is Amazon EMR). The name was once short for Elastic MapReduce.

EMR remains the choice when a team wants the open-source frameworks themselves, with control over versions, configuration and hardware. To use it well you need a picture of what a cluster is.

A cluster and its nodes

The central component is the cluster: a collection of EC2 instances, each called a node, each with a role (EMR docs: clusters and nodes).

The split between core and task nodes exists for a practical reason. Core nodes hold data, so losing one is costly. Task nodes hold none, so they can come and go. That makes task nodes the natural home for Spot Instances, spare EC2 capacity sold at a discount that AWS can reclaim. The architecture page explains that EMR runs the application master, the process that controls a job, only on core nodes, so a running job does not fail when Spot task nodes are taken away (EMR docs: architecture).

On top of the nodes, YARN manages resources by default, and you choose which applications to install at launch: Spark, Hive, Presto, HBase and others. Spark vs Hadoop covers HDFS, YARN and MapReduce; Apache Spark explained covers the engine most EMR clusters now run.

Where the data lives

A cluster has two file systems, and choosing the wrong one loses data.

• HDFS, on the cluster's own disks. It is fast, and the guide is blunt about its nature: HDFS is ephemeral storage that is reclaimed when you terminate a cluster.
• Amazon S3, addressed with s3:// paths. It exists independently of any cluster.

The guide's summary of normal practice: most often S3 is used to store input and output data, and intermediate results are stored in HDFS. Read from S3, write results to S3, and treat everything on the cluster as temporary. Then the cluster can be terminated the moment the work is done, and compute and storage scale separately.

Submitting work

There are three ways to give a cluster something to do:

• Steps. A step is a unit of work, such as a Spark application, submitted through the EMR API. Steps run in order.
• Direct submission. Connect to the primary node and use spark-submit or the application's own interface.
• Notebooks and interactive tools, for exploration.

Steps are the form to automate. The following command, adapted from the documentation and not run, creates a cluster, runs one PySpark script from S3, and shuts down:

aws emr create-cluster \
  --name "nightly-trips" \
  --release-label emr-7.14.0 \
  --applications Name=Spark \
  --instance-type m5.xlarge \
  --instance-count 3 \
  --use-default-roles \
  --auto-terminate \
  --steps Type=Spark,Name="Trips to curated",ActionOnFailure=TERMINATE_CLUSTER,Args=[--deploy-mode,cluster,s3://amzn-s3-demo-bucket/scripts/trips_to_curated.py]

The release label fixes the versions of every framework together, which is a large part of what EMR manages for you.

Transient or long-running

The command above creates a transient cluster: it exists for one job. The alternative is a long-running cluster that waits for work.

For scheduled batch work, transient clusters are usually the cheaper and cleaner design, and they are only possible because the data is in S3. Managed scaling, which adds and removes instances according to workload, reduces waste on clusters that must stay up.

Three ways to deploy

"EMR" now covers three deployment options with the same frameworks underneath:

• EMR on EC2. The classic cluster described above. Most control, most to manage.
• EMR Serverless. No cluster to configure. You create an application, submit jobs, and the service determines the resources each job needs, obtains them and releases them when the job finishes (EMR docs: EMR Serverless).
• EMR on EKS. Runs the frameworks on an Amazon EKS Kubernetes cluster, so Spark jobs share infrastructure with other containerised workloads (EMR docs: EMR on EKS).

A team starting today with batch Spark jobs and no Kubernetes platform will often find EMR Serverless the simplest entry, with EC2 clusters reserved for cases that need specific hardware, custom software on the nodes, or frameworks Serverless does not offer.

What decides the bill

EMR is billed per second with a one-minute minimum. On EC2 clusters the EMR charge is added to the price of the underlying EC2 instances and any EBS volumes (AWS: EMR pricing). The pricing page makes a point worth remembering: a 10-node cluster running for 10 hours costs the same as a 100-node cluster running for one hour. For work that parallelises well, a larger cluster for a shorter time costs the same and finishes sooner.

The practical levers:

• Terminate when idle. An idle cluster costs exactly what a busy one does.
• Spot for task nodes. Keep primary and core nodes on On-Demand capacity.
• Right-size. Run a representative job and watch utilisation before fixing the cluster size.
• Keep data in S3, in Parquet, partitioned, so jobs read less.

EMR, Glue or Athena?

AWS's own guidance is that EMR suits custom code on big data frameworks with full control over the cluster and its software. Glue vs EMR vs Lambda for ETL goes through the trade-offs for pipelines.

Mistakes to avoid

• Storing results in HDFS on a cluster that will be terminated.
• Leaving a cluster running after a job or an experiment.
• Spot Instances for core nodes, putting HDFS data at risk.
• One permanent cluster shared by everything, so one job's configuration needs fight another's.
• Choosing EMR by habit for a job Glue or Athena would run with no cluster at all.

The Hadoop ecosystem, Apache Spark and Amazon EMR are covered in the Data Engineering course.

Sources

• What is Amazon EMR? (EMR Management Guide) — AWS Documentation
• Understanding how to create and work with Amazon EMR clusters (EMR Management Guide) — AWS Documentation
• Amazon EMR architecture and service layers (EMR Management Guide) — AWS Documentation
• What is Amazon EMR Serverless? (EMR Serverless User Guide) — AWS Documentation
• What is Amazon EMR on EKS? (EMR on EKS Development Guide) — AWS Documentation
• Amazon EMR pricing (Amazon Web Services) — Amazon Web Services
• Add a Spark step (EMR Release Guide) — AWS Documentation