Partitioning and Bucketing in Spark Explained

In Spark, "partitioning" means three different things. People mix them up, and the fix for a slow job depends on which one is at fault. In-memory partitions decide how work is split across tasks while a job runs. Directory partitioning (partitionBy) decides how files are laid out on disk, so later queries can skip them. Bucketing (bucketBy) pre-organises a table by the hash of a column, so later joins and aggregations on that column need less shuffling. This guide takes them one at a time.

1. In-memory partitions: how work is split

Each Spark task processes one partition of a DataFrame. When Spark reads files, it packs input into partitions of up to spark.sql.files.maxPartitionBytes, 128 MB by default. After a shuffle, which happens for joins and aggregations, the number of partitions comes from spark.sql.shuffle.partitions, 200 by default (Spark: performance tuning). Too few partitions and you cannot use all your cores; too many tiny ones and scheduling overhead dominates.

Adaptive Query Execution, enabled by default since Spark 3.2.0, uses runtime statistics to coalesce small post-shuffle partitions, so the fixed 200 matters less than it used to. You still control partitioning explicitly with two methods, which behave very differently:

The PySpark reference gives the trade-off. coalesce from 1,000 to 100 partitions avoids a shuffle, because each new partition simply claims 10 old ones. But a drastic coalesce, down to one partition for example, can push the whole upstream computation onto very few tasks. In that case repartition is better, despite the shuffle (PySpark: coalesce).

2. Directory partitioning with partitionBy

partitionBy writes one sub-directory per value of the partition column, named column=value (Spark: bucketing, sorting and partitioning). When a later query filters on that column, Spark reads only the matching directories. Reading the table back, Spark discovers the partition columns from the paths and infers their types (Spark: partition discovery).

(events
    .write
    .partitionBy("event_date")
    .mode("overwrite")
    .parquet("s3://my-lake/curated/events/"))

# Resulting layout:
#   s3://my-lake/curated/events/event_date=2026-09-01/part-00000-….parquet
#   s3://my-lake/curated/events/event_date=2026-09-02/part-00000-….parquet

daily = (spark.read.parquet("s3://my-lake/curated/events/")
              .where("event_date = '2026-09-02'"))    # reads one directory, not all

Choose the partition column by one test: do most queries filter on it, and does it have a modest number of values? Dates are the classic choice. The Spark documentation warns that because partitionBy creates a directory per value, it has "limited applicability to columns with high cardinality". Partition by customer_id and you get one directory per customer, each holding tiny files.

Overwriting only some partitions

Re-running yesterday's load with mode("overwrite") on a partitioned table is a well-known trap. In the default static mode, Spark deletes all partitions matching the partition specification before writing, which for a DataFrame write can mean the whole table. In dynamic mode, it replaces only the partitions the job actually writes data into (Spark: configuration):

(yesterday
    .write
    .option("partitionOverwriteMode", "dynamic")   # or set spark.sql.sources.partitionOverwriteMode
    .partitionBy("event_date")
    .mode("overwrite")
    .parquet("s3://my-lake/curated/events/"))

3. Bucketing with bucketBy

Bucketing splits a table into a fixed number of files by the hash of a column. Rows with the same key always land in the same bucket, however many distinct keys there are, which is why bucketing works for high-cardinality columns where partitionBy does not. Bucketing and sorting apply only to persistent tables: you write with saveAsTable, not save or parquet(path) (Spark: bucketing, sorting and partitioning).

(orders
    .write
    .bucketBy(16, "customer_id")
    .sortBy("customer_id")
    .mode("overwrite")
    .saveAsTable("sales.orders_bucketed"))

(customers
    .write
    .bucketBy(16, "customer_id")
    .sortBy("customer_id")
    .mode("overwrite")
    .saveAsTable("sales.customers_bucketed"))

joined = spark.table("sales.orders_bucketed").join(
    spark.table("sales.customers_bucketed"), "customer_id")
joined.explain()   # check the plan: no Exchange before the join means the shuffle was avoided

The payoff comes when both sides of a join are bucketed the same way on the join key. Spark can then use the existing layout instead of shuffling both tables. The tuning guide calls these bucket joins, and treats the newer storage-partitioned join as their generalisation (Spark: performance tuning). Always confirm with explain(): bucket counts that differ between tables, or a join key that is not the bucket column, bring the shuffle back.

You can combine both layouts, for example partitionBy("event_date").bucketBy(16, "customer_id"): dates for pruning, buckets for joins within each date.

Choosing between them

Shuffles, join strategies and caching are covered in Spark performance tuning: shuffles, joins and caching. If DataFrame basics such as transformations, actions and lazy evaluation are new, start with the PySpark tutorial.

Where to go next

Spark architecture, DataFrames, Spark SQL and optimisation are part of the Apache Spark module in our Data Engineering course.

Sources

• Generic Load/Save Functions: bucketing, sorting and partitioning — Apache Spark documentation
• Parquet Files: partition discovery — Apache Spark documentation
• Performance Tuning — Apache Spark documentation
• Spark Configuration — Apache Spark documentation
• DataFrame.coalesce and DataFrame.repartition — PySpark API reference