Apache Spark Explained for Beginners

Apache Spark is, in its own documentation's words, a unified analytics engine for large-scale data processing (Spark docs: overview). In practice it lets you write one program, in Python, SQL, Scala or R, and run it across many machines, each working on part of the data. This guide explains how that works, shows Spark's most important behaviour on three million real taxi trips, and is honest about when you do not need Spark at all.

When Spark is the right tool

Spark earns its complexity when data is too big for one machine's memory, when a job would take hours on one machine, or when a team already runs its pipelines on a Spark platform such as Databricks, Amazon EMR or Google Dataproc. For a file of a few hundred megabytes on a laptop, pandas, Polars or DuckDB is usually simpler and faster; PySpark vs pandas compares them. The example below is deliberately small so you can run it anywhere; the concepts are identical at a thousand times the size.

How Spark runs a program

Spark's cluster overview describes a small set of moving parts (Spark docs: cluster overview):

• Driver: the process running your program's main function. It builds the plan and schedules work.
• Executors: processes on worker machines that run tasks and hold data in memory or on disk. Each application gets its own.
• Cluster manager: allocates machines. Spark supports its own standalone manager, Hadoop YARN and Kubernetes.
• Job, stage, task: an action, such as writing or collecting results, triggers a job; the job is split into stages; each stage runs as tasks, one per partition of data.

In local mode, used below, the driver and executor share one process on your machine, with worker threads standing in for executors. It is how most people learn Spark and test code before running it on a cluster.

Lazy evaluation, seen on real data

The single most important idea in Spark is laziness. Transformations, such as filter, group or join, are not computed right away; Spark remembers them and computes only when an action needs a result (Spark docs: programming guide). You can watch this happen by counting the jobs Spark has run. The data is January 2024 New York yellow taxi trips, published by the city's Taxi and Limousine Commission as Parquet files with a zone lookup table (NYC TLC: trip record data):

import urllib.request
from pathlib import Path
from pyspark.sql import SparkSession, functions as F

base = "https://d37ci6vzurychx.cloudfront.net"
for name, url in {"yellow_tripdata_2024-01.parquet": f"{base}/trip-data/yellow_tripdata_2024-01.parquet",
                  "taxi_zone_lookup.csv": f"{base}/misc/taxi_zone_lookup.csv"}.items():
    if not Path(name).exists():
        urllib.request.urlretrieve(url, name)

spark = SparkSession.builder.master("local[4]").appName("taxi").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
jobs = lambda: len(spark.sparkContext.statusTracker().getJobIdsForGroup())

trips = spark.read.parquet("yellow_tripdata_2024-01.parquet")
print("jobs after reading the Parquet file:", jobs())
zones = spark.read.csv("taxi_zone_lookup.csv", header=True, inferSchema=True)
print("jobs after reading the CSV with inferSchema:", jobs())
busiest = (trips.where(F.col("fare_amount") >= 0)
                .groupBy("PULocationID")
                .agg(F.count("*").alias("trips"), F.round(F.avg("fare_amount"), 2).alias("avg_fare"))
                .join(zones, F.col("PULocationID") == F.col("LocationID"))
                .select("Zone", "Borough", "trips", "avg_fare")
                .orderBy(F.desc("trips")))
print("jobs after transformations:", jobs())
busiest.show(5, truncate=False)
print("jobs after show():", jobs())
print("input partitions:", trips.rdd.getNumPartitions())
spark.stop()

jobs after reading the Parquet file: 1
jobs after reading the CSV with inferSchema: 3
jobs after transformations: 3
+---------------------+---------+------+--------+
|Zone                 |Borough  |trips |avg_fare|
+---------------------+---------+------+--------+
|Midtown Center       |Manhattan|141749|15.58   |
|Upper East Side South|Manhattan|141270|12.42   |
|JFK Airport          |Queens   |141196|62.46   |
|Upper East Side North|Manhattan|135339|12.92   |
|Midtown East         |Manhattan|105469|15.14   |
+---------------------+---------+------+--------+
only showing top 5 rows

jobs after show(): 6
input partitions: 4

Read the job counts in order:

• Reading Parquet ran one small job, to read the schema stored in the file. Parquet files carry their own schema, so Spark did not scan the data.
• Reading the CSV ran two more. A CSV has no stored types, and inferSchema requires one extra pass over the data (Spark docs: CSV). On a 12 KB lookup file that is trivial; on a 500 GB CSV it is an expensive full scan before any real work, which is why production code declares schemas.
• Filter, group, join, select and sort ran nothing. The count stayed at three: Spark only built a plan.
• show() triggered the work, three jobs covering the scan, aggregation, join and sort, because it needed rows to display.

Laziness is what lets Spark optimise: it sees the whole chain before running any of it, so it can push the fare filter into the Parquet scan and read only the columns it needs. The result is also a useful sanity check on the data: JFK Airport has almost as many pickups as Midtown, at four times the average fare.

Transformations and actions

A practical consequence: an error in a transformation often appears only when an action runs, several lines later. And each action recomputes the plan from the source unless you cache an intermediate result, which the programming guide describes with persist and cache.

Partitions: the unit of parallelism

Spark splits data into partitions and runs one task per partition. The taxi file became four input partitions, one per thread of local[4]. On a cluster, the number and size of partitions decide how evenly work spreads across executors, and poor partitioning is behind many slow jobs; partitioning and bucketing in Spark covers how to control it.

Getting started

• Install: pip install pyspark gives you local mode. Spark 3.5 runs on Java 8, 11 or 17; the Spark 4 releases require Java 17 or later, so check your Java version first.
• Use DataFrames and Spark SQL, not the older RDD API, for almost all work; they are optimised automatically.
• Learn the API with the PySpark tutorial, then managed platforms with the Databricks tutorial.

Mistakes beginners make

• collect() on large data, which pulls everything to the driver and can crash it.
• inferSchema on big files, an extra full scan; declare the schema instead.
• Python UDFs for things built-in functions do, which move data between the JVM and Python and are usually much slower.
• Using Spark for small data, paying cluster start-up and coordination costs for a job pandas finishes in seconds.

When jobs get slow, Spark performance tuning covers shuffles, joins and caching, and Spark vs Hadoop explains where Spark came from.

Spark architecture, PySpark basics, DataFrames and RDDs, Spark SQL and performance optimization are part of the Apache Spark module of the Data Engineering course.

Sources

• Spark 3.5.7 overview (Apache Spark documentation) — Apache Spark
• Cluster mode overview (Apache Spark documentation) — Apache Spark
• RDD programming guide (Apache Spark documentation) — Apache Spark
• CSV files (Spark SQL documentation) — Apache Spark
• TLC Trip Record Data (NYC Taxi and Limousine Commission) — NYC Taxi and Limousine Commission