Spark vs Hadoop: What Changed and Why

"Spark vs Hadoop" is a slightly misleading contest, because Hadoop is not one thing. It is a storage system, a resource manager and a processing engine, and Spark replaced only the last of those. Understanding which part changed, and why, explains a great deal about how today's data platforms look.

Hadoop's three parts

• HDFS is a distributed file system for commodity hardware, built for fault tolerance and throughput rather than low latency. A NameNode tracks the namespace, DataNodes store files as blocks, typically 128 MB, replicated three times by default, and files are written once and read many times (Hadoop docs: HDFS).
• YARN manages cluster resources: a global ResourceManager allocates capacity, a per-application ApplicationMaster schedules that application's work, and a NodeManager on each machine runs containers (Hadoop docs: YARN).
• MapReduce is the processing framework: a job splits its input into independent chunks for map tasks, sorts their output, and feeds it to reduce tasks, with input and output stored in the file system and tasks scheduled on the machines that hold the data (Hadoop docs: MapReduce).

The MapReduce model, step by step

The model is easiest to understand by running it. This plain Python emulation counts taxi pickups per zone in January 2024 New York yellow taxi data (NYC TLC: trip record data) the MapReduce way: map tasks emit key and value pairs to partitioned files on disk, then reducers read, sort and sum them. It is a teaching model, not Hadoop, but the phases are the same:

import os
import urllib.request
from collections import Counter
from itertools import groupby
from pathlib import Path
import pyarrow.parquet as pq

name = "yellow_tripdata_2024-01.parquet"
if not Path(name).exists():
    urllib.request.urlretrieve(f"https://d37ci6vzurychx.cloudfront.net/trip-data/{name}", name)
source = pq.ParquetFile(name)
REDUCERS = 2
os.makedirs("shuffle", exist_ok=True)

# map phase: one map task per input split (here, a Parquet row group); each writes (key, 1) pairs to disk
for split in range(source.num_row_groups):
    zones = source.read_row_group(split, columns=["PULocationID"]).column(0).to_pylist()
    files = [open(f"shuffle/map{split}_part{r}.txt", "w", newline="\n") for r in range(REDUCERS)]
    for zone in zones:
        files[zone % REDUCERS].write(f"{zone}\t1\n")            # partition by key
    for f in files:
        f.close()

spilled = sum(os.path.getsize(os.path.join("shuffle", f)) for f in os.listdir("shuffle"))
print(f"map tasks: {source.num_row_groups}, intermediate data written to disk: {spilled / 1e6:.1f} MB")

# reduce phase: each reducer reads its partition from every map task, sorts by key, and sums
totals = Counter()
for r in range(REDUCERS):
    pairs = []
    for split in range(source.num_row_groups):
        with open(f"shuffle/map{split}_part{r}.txt") as f:
            pairs += [line.rstrip("\n").split("\t") for line in f]
    pairs.sort(key=lambda kv: int(kv[0]))
    for zone, group in groupby(pairs, key=lambda kv: int(kv[0])):
        totals[zone] = sum(int(v) for _, v in group)
print("top zones:", totals.most_common(3))
print("total trips:", sum(totals.values()))

map tasks: 3, intermediate data written to disk: 17.3 MB
top zones: [(132, 145240), (161, 143471), (237, 142708)]
total trips: 2964624

Three things in this output explain Hadoop's strengths and weaknesses:

• The model scales. Map tasks are independent, so a thousand machines can each map a thousand splits; partitioning by key sends every pickup for zone 132, JFK Airport, to the same reducer, so each reducer's answer is complete for its keys.
• Everything goes through disk. Counting a single 50 MB file wrote 17.3 MB of intermediate data before any reducer started. On a cluster, map output is written to local disks, shuffled over the network, and final output written to HDFS with three replicas.
• One job is one map and one reduce. A pipeline that joins, filters, aggregates and ranks is a chain of MapReduce jobs, each writing its full result to HDFS for the next to read.

That last point is what made iterative work slow. A machine learning algorithm that passes over the same data twenty times paid twenty rounds of disk reads and writes.

What Spark changed

Spark began as research at UC Berkeley. The paper introducing its core abstraction, resilient distributed datasets, targeted exactly the workloads MapReduce handled poorly, iterative algorithms and interactive data mining, observing that keeping data in memory can improve performance by an order of magnitude (NSDI 2012 paper). Three changes followed:

• In-memory processing. Intermediate results can stay in executor memory, cached across steps, instead of being written to HDFS between jobs.
• A graph of stages, not a chain of jobs. Spark plans a whole pipeline as one job divided into stages at shuffle boundaries, stages that its documentation compares to MapReduce's map and reduce phases (Spark docs: cluster overview). Filters and projections within a stage run together without touching disk.
• Higher-level APIs. DataFrames, SQL, streaming and machine learning libraries replaced hand-written mapper and reducer classes. The same count above is one line in Spark: trips.groupBy("PULocationID").count().

Fault tolerance changed too. Instead of relying on replicated intermediate files, Spark records how each dataset was derived and recomputes lost partitions from that lineage.

What Spark still takes from Hadoop

Spark did not replace the whole stack:

• YARN remains one of Spark's cluster managers, alongside Spark's standalone manager and Kubernetes.
• HDFS and Hadoop's file system libraries are still how Spark talks to storage, including cloud object stores reached through Hadoop connectors.
• Formats from the Hadoop ecosystem, Parquet and ORC, are still the standard for analytical files; Parquet vs ORC vs Avro compares them.
• The Hive metastore pattern of table catalogs persists in many platforms.

What this means today

New platforms rarely run MapReduce jobs, and many no longer run HDFS: cloud object storage has taken much of its role, with Spark on managed services or Kubernetes doing the processing. Existing on-premises Hadoop clusters still run in many organisations, often with Spark on YARN, so knowing both remains useful. For a new data engineer, the priority is clear: learn Spark well, starting with Apache Spark explained, and learn enough about HDFS, YARN and MapReduce to understand the systems Spark grew out of. Distributed systems basics covers the ideas both share: partitioning, replication and failure.

The Hadoop ecosystem, distributed computing concepts and Apache Spark are covered in the big data fundamentals and Spark modules of the Data Engineering course.

Sources

• MapReduce tutorial (Apache Hadoop documentation) — Apache Hadoop
• HDFS architecture (Apache Hadoop documentation) — Apache Hadoop
• Apache Hadoop YARN (Apache Hadoop documentation) — Apache Hadoop
• Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing (NSDI 2012) — USENIX NSDI 2012
• Cluster mode overview (Apache Spark documentation) — Apache Spark
• TLC Trip Record Data (NYC Taxi and Limousine Commission) — NYC Taxi and Limousine Commission