Apache Spark Interview Questions and Answers

Apache Spark interview questions often get textbook answers: "transformations are lazy", "groupBy causes a shuffle", "small tables are broadcast". Interviewers then ask how you know, and how you would check in your own job. This article answers seven core questions with evidence from a real Spark session: how many jobs ran, what the physical plan contains, how many partitions came out, and how rows were distributed.

The data is one month of New York City yellow taxi trips, about three million rows, plus the 265-row zone lookup table, both from the TLC (NYC TLC: trip record data). Spark runs locally with four cores, which is enough to show how it plans and executes work.

from pyspark.sql import SparkSession, functions as F

spark = (SparkSession.builder.master("local[4]").config("spark.ui.showConsoleProgress", "false")
         .config("spark.driver.memory", "2g").getOrCreate())
spark.sparkContext.setLogLevel("ERROR")
tracker = spark.sparkContext.statusTracker()


def jobs_run(group, action):                                   # count the Spark jobs an action launches
    spark.sparkContext.setJobGroup(group, group)
    result = action()
    return result, len(tracker.getJobIdsForGroup(group))


def plan(df):                                                  # the physical plan as text
    return df._jdf.queryExecution().executedPlan().toString()


trips = spark.read.parquet("yellow_tripdata_2024-01.parquet")   # NYC TLC yellow taxi trips, January 2024
zones = spark.read.option("header", True).csv("taxi_zone_lookup.csv")

print("S1 lazy evaluation")
_, before = jobs_run("define", lambda: trips.filter(F.col("trip_distance") > 10).select("PULocationID", "total_amount"))
long_trips = trips.filter(F.col("trip_distance") > 10).select("PULocationID", "total_amount")
n, after = jobs_run("count", long_trips.count)
print(f"   jobs while defining the query: {before} | jobs for count(): {after} | trips over 10 miles: {n:,}")

print("\nS2 narrow vs wide transformations (is there a shuffle 'Exchange' in the plan?)")
print("   filter + select:", "Exchange" in plan(long_trips), "| groupBy + count:", "Exchange" in plan(trips.groupBy("PULocationID").count()))

print("\nS3 partitions")
by_zone = trips.groupBy("PULocationID").agg(F.sum("total_amount").alias("revenue"))
print(f"   input partitions {trips.rdd.getNumPartitions()} | spark.sql.shuffle.partitions {spark.conf.get('spark.sql.shuffle.partitions')} "
      f"| partitions after the groupBy with adaptive execution: {by_zone.rdd.getNumPartitions()}")
spark.conf.set("spark.sql.adaptive.enabled", "false")
print(f"   the same groupBy with adaptive execution off: {trips.groupBy('PULocationID').agg(F.sum('total_amount')).rdd.getNumPartitions()} partitions")
spark.conf.set("spark.sql.adaptive.enabled", "true")

print("\nS4 join strategies")
joined = trips.join(zones, trips.PULocationID == zones.LocationID)
print("   default plan uses BroadcastHashJoin:", "BroadcastHashJoin" in plan(joined))
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")
print("   with broadcasting disabled, SortMergeJoin:", "SortMergeJoin" in plan(trips.join(zones, trips.PULocationID == zones.LocationID)))
spark.conf.unset("spark.sql.autoBroadcastJoinThreshold")

print("\nS5 skew: rows per partition after repartitioning by pickup zone")
sizes = sorted(trips.repartition(8, "PULocationID").rdd.glom().map(len).collect())
print(f"   8 partitions: smallest {sizes[0]:,} rows, largest {sizes[-1]:,} rows ({sizes[-1] / (sum(sizes) / 8):.1f}x an even share)")

print("\nS6 caching: later queries read the cached data")
cached = long_trips.cache()
cached.count()                                                  # the first action fills the cache
print(f"   is_cached: {cached.is_cached} | storage level: {cached.storageLevel}")
print("   the cached plan contains InMemoryTableScan:", "InMemoryTableScan" in plan(cached.groupBy("PULocationID").count()))

print("\nS7 DataFrame vs RDD give the same answer")
df_top = trips.groupBy("PULocationID").count().orderBy(F.desc("count")).first()
rdd_top = trips.select("PULocationID").rdd.map(lambda r: (r[0], 1)).reduceByKey(lambda a, b: a + b).max(key=lambda kv: kv[1])
print(f"   DataFrame: zone {df_top['PULocationID']} with {df_top['count']:,} pickups | RDD: zone {rdd_top[0]} with {rdd_top[1]:,}")
spark.stop()

S1 lazy evaluation
   jobs while defining the query: 0 | jobs for count(): 2 | trips over 10 miles: 228,254

S2 narrow vs wide transformations (is there a shuffle 'Exchange' in the plan?)
   filter + select: False | groupBy + count: True

S3 partitions
   input partitions 4 | spark.sql.shuffle.partitions 200 | partitions after the groupBy with adaptive execution: 1
   the same groupBy with adaptive execution off: 200 partitions

S4 join strategies
   default plan uses BroadcastHashJoin: True
   with broadcasting disabled, SortMergeJoin: True

S5 skew: rows per partition after repartitioning by pickup zone
   8 partitions: smallest 96,742 rows, largest 574,147 rows (1.5x an even share)

S6 caching: later queries read the cached data
   is_cached: True | storage level: Disk Memory Deserialized 1x Replicated
   the cached plan contains InMemoryTableScan: True

S7 DataFrame vs RDD give the same answer
   DataFrame: zone 132 with 145,240 pickups | RDD: zone 132 with 145,240

S1. What does lazy evaluation mean in practice?

Transformations such as filter and select only record what to do; nothing runs until an action such as count needs a result (Spark: RDD programming guide). Defining the query launched 0 jobs; calling count() launched 2 and returned 228,254 trips over 10 miles. Laziness lets Spark optimise the whole chain before running it, for example by reading only the columns it needs. It also explains a common surprise: an error in a transformation only appears when an action runs, sometimes many lines later.

S2. What is the difference between narrow and wide transformations?

A narrow transformation, such as a filter or a projection, works on each partition independently. A wide one, such as groupBy or most joins, needs rows with the same key together, so Spark shuffles data across partitions. The physical plan shows it: the filter-and-select plan had no Exchange operator, and the groupBy plan had one. Shuffles mean network and disk work, so in an interview, point at the Exchange operators when asked where a job spends its time.

S3. How many partitions does a Spark job have?

It depends on the stage. Reading the Parquet file gave 4 partitions. After a shuffle, Spark uses spark.sql.shuffle.partitions, 200 by default, which is far too many for a small result: 200 tiny tasks. Adaptive Query Execution, on by default since Spark 3.2, merges small post-shuffle partitions using runtime statistics (Spark: performance tuning). Here it reduced the grouped result to 1 partition; with AQE switched off, the same query kept all 200.

S4. When does Spark broadcast a join?

When one side is small enough: under spark.sql.autoBroadcastJoinThreshold, 10 MB by default. Joining trips to the zone table produced a BroadcastHashJoin: the zone table is copied to every executor and the large table is never shuffled. With broadcasting disabled by setting the threshold to -1, the same join became a SortMergeJoin, which shuffles and sorts both sides. Know how to read which one you got, and know the broadcast hint for when statistics are missing.

S5. What is data skew, and how do you see it?

Uneven distribution of rows across partitions, usually because some keys are far more common than others. Repartitioning by pickup zone into 8 partitions gave a smallest partition of 96,742 rows and a largest of 574,147, 1.5 times an even share. The busiest partition sets the stage's duration, because every other task finishes and waits. Remedies include AQE's skew-join handling, salting hot keys, or choosing a different key.

S6. When should you cache?

When the same intermediate result is used by several actions, and it fits comfortably in memory. After cache() and a first action, the DataFrame reported is_cached as true with a memory-and-disk storage level, and a later query's plan read it through InMemoryTableScan instead of rescanning the Parquet file. Caching something used once only costs memory; unpersist when you are done.

S7. DataFrame or RDD?

Prefer DataFrames. Both found the busiest pickup zone, 132 with 145,240 pickups, but the DataFrame version goes through Spark's optimiser and runs in the JVM, while the RDD version passes every row through Python lambdas. RDDs remain useful for unstructured data and fine-grained control, but in interviews, the expected default is the DataFrame or SQL API.

One more: driver, executors, jobs, stages and tasks

The driver runs your program, builds the plan and schedules work; executors run tasks and hold cached data. Each action becomes one or more jobs, each job is split into stages at shuffle boundaries, and each stage runs one task per partition. One action can therefore launch more than one job, as the single count() in S1 did, and the number of partitions decides how much parallel work each stage can do. With adaptive execution, Spark also runs query stages one after another so that it can re-plan using the statistics of the stages already finished.

How to answer Spark questions

• Point to evidence: the Spark UI, explain(), job counts and partition sizes.
• Name the expensive parts: shuffles, skewed partitions, Python round trips.
• Mention the version: AQE changed many classic tuning answers from Spark 3.0 onwards.

Performance work continues in Spark performance interview questions and the Spark tuning guide on shuffles, joins and caching. For API-level questions, see PySpark interview questions; for when Spark is the wrong tool, PySpark vs pandas; and for the full round, data engineering interview questions and answers.

Apache Spark is taught in the Data Engineering course.

Sources

• RDD Programming Guide (Apache Spark 3.5 documentation) — The Apache Software Foundation
• Performance Tuning (Apache Spark 3.5 documentation) — The Apache Software Foundation
• TLC Trip Record Data (NYC Taxi and Limousine Commission) — NYC Taxi and Limousine Commission