Spark Performance Interview Questions
Spark performance questions ask what you would change, and how you would know it helped. Five tuning levers are tested on 3.5 million real taxi trips with evidence that does not depend on the machine: pushed-down filters in the plan, task counts, partition sizes before and after salting, join types, and cache hits.
GloryTecks Team
Training & Placement Faculty
Last updated: 9 October 2026
Spark performance interview questions are rarely "what is a shuffle?". They are "this job is slow; what would you change, and how would you know it helped?". A stopwatch is a weak answer, because timings change with the machine, the data and whatever else is running. This article tests five tuning levers on 3.5 million real New York taxi trips from January 2025 (NYC TLC: trip record data), and backs each with evidence that stays the same on any machine: what the plan contains, how many tasks ran, how rows were spread, and whether queries hit the cache. Timings are shown too, but marked as varying.
import time
from pyspark.sql import SparkSession, functions as F
spark = (SparkSession.builder.master("local[4]").config("spark.ui.showConsoleProgress", "false")
.config("spark.driver.memory", "3g").getOrCreate())
spark.sparkContext.setLogLevel("ERROR")
tracker = spark.sparkContext.statusTracker()
trips = spark.read.parquet("yellow_tripdata_2025-01.parquet") # NYC TLC yellow taxi trips, January 2025
plan = lambda df: df._jdf.queryExecution().executedPlan().toString()
def timed(df):
start = time.perf_counter()
df.write.format("noop").mode("overwrite").save() # run the whole query without writing output
return time.perf_counter() - start
def tasks(group, df): # total tasks across the stages a query ran
spark.sparkContext.setJobGroup(group, group)
df.write.format("noop").mode("overwrite").save()
stages = [s for j in tracker.getJobIdsForGroup(group) for s in tracker.getJobInfo(j).stageIds]
return sum(tracker.getStageInfo(s).numTasks for s in stages if tracker.getStageInfo(s))
print("SP1 column pruning and predicate pushdown into Parquet")
q = trips.filter(F.col("trip_distance") > 20).select("PULocationID", "total_amount")
scan = next(line for line in plan(q).splitlines() if "FileScan" in line)
print(" reads only the needed columns:", "ReadSchema: struct<trip_distance:double,PULocationID:int,total_amount:double>" in scan,
"| filter pushed to the reader:", "PushedFilters: [IsNotNull(trip_distance), GreaterThan(trip_distance,20.0)]" in scan)
print("\nSP2 shuffle partitions with adaptive execution off")
spark.conf.set("spark.sql.adaptive.enabled", "false")
agg = lambda: trips.groupBy("PULocationID", "DOLocationID").agg(F.sum("total_amount"))
for n in (200, 8):
spark.conf.set("spark.sql.shuffle.partitions", str(n))
print(f" shuffle.partitions={n:<3} -> {tasks(f'p{n}', agg())} tasks in total, {timed(agg()):.1f} s (varies)")
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.shuffle.partitions", "8")
print("\nSP3 a skewed key, and salting it")
by_zone = trips.repartition(8, "PULocationID").rdd.glom().map(len).collect()
salted = trips.withColumn("salt", (F.rand(seed=1) * 8).cast("int")).repartition(8, "PULocationID", "salt").rdd.glom().map(len).collect()
for name, sizes in (("by zone", by_zone), ("by zone + salt", salted)):
print(f" {name:<15} largest partition {max(sizes):>9,} rows, {max(sizes) / (sum(sizes) / 8):.2f}x an even share")
print("\nSP4 broadcasting a larger dimension table")
dims = (trips.groupBy("PULocationID", "DOLocationID").agg(F.avg("trip_distance").alias("avg_miles"))
.withColumnRenamed("PULocationID", "pu").withColumnRenamed("DOLocationID", "do"))
dims = spark.createDataFrame(dims.collect()).cache()
print(f" route table: {dims.count():,} rows")
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")
cond = (trips.PULocationID == dims.pu) & (trips.DOLocationID == dims.do)
plain, hinted = trips.join(dims, cond), trips.join(F.broadcast(dims), cond)
print(" without a hint: SortMergeJoin", "SortMergeJoin" in plan(plain), "| with broadcast(): BroadcastHashJoin", "BroadcastHashJoin" in plan(hinted))
print(f" time: sort-merge {timed(plain):.1f} s, broadcast {timed(hinted):.1f} s (varies)")
spark.conf.unset("spark.sql.autoBroadcastJoinThreshold")
print("\nSP5 reusing a result: scans of the Parquet file with and without cache")
base = trips.filter(F.col("total_amount") > 0)
for label in ("no cache", "cached"):
df = base.cache() if label == "cached" else base # cache only on the second pass
df.count()
reports = [df.groupBy("payment_type").count(), df.agg(F.avg("tip_amount")), df.agg(F.max("trip_distance"))]
plans = [plan(r) for r in reports]
from_file = sum("InMemoryTableScan" not in p for p in plans) # a cached plan still describes its source, so test the scan type
print(f" {label:<9} follow-up queries reading the Parquet file: {from_file} of 3 | reading the cache: {3 - from_file} of 3")
spark.stop()SP1 column pruning and predicate pushdown into Parquet
reads only the needed columns: True | filter pushed to the reader: True
SP2 shuffle partitions with adaptive execution off
shuffle.partitions=200 -> 204 tasks in total, 2.5 s (varies)
shuffle.partitions=8 -> 12 tasks in total, 0.5 s (varies)
SP3 a skewed key, and salting it
by zone largest partition 658,599 rows, 1.52x an even share
by zone + salt largest partition 512,199 rows, 1.18x an even share
SP4 broadcasting a larger dimension table
route table: 32,797 rows
without a hint: SortMergeJoin True | with broadcast(): BroadcastHashJoin True
time: sort-merge 6.2 s, broadcast 1.2 s (varies)
SP5 reusing a result: scans of the Parquet file with and without cache
no cache follow-up queries reading the Parquet file: 3 of 3 | reading the cache: 0 of 3
cached follow-up queries reading the Parquet file: 0 of 3 | reading the cache: 3 of 3SP1. How do you make Spark read less data?
Select only the columns you need and filter early, then check the plan. For a query using three columns and a distance filter, the Parquet scan's ReadSchema listed exactly those three columns, and PushedFilters contained the distance condition. The reader can therefore skip whole row groups whose statistics rule them out (Spark: Parquet). Reading less is the cheapest optimisation there is. Pushdown breaks when the filter wraps the column in a function or applies a Python UDF, so check the plan rather than assuming.
SP2. How should you set shuffle partitions?
To fit the data, not the default. With adaptive execution off, an aggregation used 204 tasks with the default 200 shuffle partitions, and 12 with 8. Each task has fixed overhead, so hundreds of tiny tasks on a small result waste time: here 2.5 seconds against 0.5 on this laptop, a difference that will vary by machine. Adaptive execution's own target for a coalesced partition, spark.sql.adaptive.advisoryPartitionSizeInBytes, defaults to 64 MB, a useful order of magnitude for sizing partitions. Adaptive execution coalesces small partitions automatically, which is why many teams now set a generous value and let AQE reduce it (Spark: performance tuning).
SP3. How do you deal with a skewed key?
First measure it, then spread it. Partitioning by pickup zone put 658,599 rows in the largest of 8 partitions, 1.52 times an even share. Adding a random salt from 0 to 7 to the key cut the largest partition to 512,199 rows, 1.18 times even. The slowest task finishes sooner, and so does the stage. The cost is that the other side of a join must be replicated once per salt value, so salt only the hot keys if you can. In Spark 3, check first whether AQE's skew-join optimisation already handles it.
SP4. When should you force a broadcast join?
When one side is small enough to copy to every executor but Spark does not know it, for example because statistics are missing or the table is just above the threshold. A route table of 32,797 rows, joined to trips with automatic broadcasting disabled, used a SortMergeJoin, which shuffles both sides. Wrapping it in broadcast() produced a BroadcastHashJoin (Spark: hints), and the join ran in 1.2 seconds instead of 6.2 on this machine. Do not broadcast something large: every executor must hold a full copy in memory.
SP5. When does caching pay off?
When several queries reuse the same expensive intermediate result. Without caching, all three follow-up queries planned a fresh read of the Parquet file; after cache() and one action to fill it, all three read from the in-memory columns instead. Caching costs memory and a first full computation, so cache results that are reused, that are expensive to recompute, and that fit. Then unpersist them.
SP6. What is the small files problem?
Thousands of tiny files make every read slow, because each file costs a listing, an open and a task, and each tiny task does almost no work. They usually come from writing with too many partitions, or from streaming jobs that write every few seconds. Fix it at write time with coalesce or repartition before saving, compact existing files on a schedule, or use a table format such as Delta Lake or Iceberg, which can compact for you.
How to answer performance questions
- Diagnose first: find the slow stage in the Spark UI, then look at its tasks: many tiny ones, one huge one, or heavy shuffle.
- Change one thing at a time, and show evidence that does not depend on the machine.
- Prefer reading less before tuning how data moves.
- Know what AQE already does in your Spark version.
Fundamentals are covered in Apache Spark interview questions and API details in PySpark interview questions; the full tuning walkthrough is in Spark performance tuning: shuffles, joins and caching. Skew also shapes stream design, as in system design interview questions for data engineers, and the full round is in data engineering interview questions and answers.
What are common Spark performance interview questions?
How do I find out why a Spark job is slow?
What is salting in Spark?
Does Adaptive Query Execution make tuning unnecessary?
Spark tuning is part of the Data Engineering course.
Sources
- Performance Tuning (Apache Spark 3.5 documentation) — The Apache Software Foundation
- Parquet Files (Apache Spark 3.5 documentation) — The Apache Software Foundation
- Hints (Apache Spark 3.5 documentation) — The Apache Software Foundation
- TLC Trip Record Data (NYC Taxi and Limousine Commission) — NYC Taxi and Limousine Commission
GloryTecks Team
Training & Placement Faculty
The GloryTecks faculty team mentors learners across Data Science, AI and Data Engineering, with a focus on real-time projects and placement readiness in Hyderabad.
Want to master Interview Questions?
Join GloryTecks’ expert-led Data Engineering training in Hyderabad with real-time projects and 100% placement support.
Behavioral Interview Questions for Data Roles
NextNLP Interview Questions for Data Scientists
Related articles
Apache Spark Interview Questions and Answers
Spark interviews test whether you can predict what Spark will do with your code. Seven core questions are answered here with evidence from a real Spark session over three million taxi trips: job counts, physical plans, partition numbers and row distributions, instead of definitions recited from memory.
SQL Interview Questions: Top 50 with Answers
Fifty SQL interview questions with answers checked on PostgreSQL, DuckDB and SQLite against one small practice schema: evaluation order, NULL logic, joins and fan-out, window functions, dates, and performance, including where the three engines disagree.
Python Interview Questions for Data Roles
The plain-Python coding tasks data-analyst and data-engineer interviews use, without pandas: counting, deduplicating, grouping, streaming large files, flattening JSON, joining, and the bugs interviewers plant. Each has a worked solution and the follow-up questions to expect.