PySpark vs Pandas: When to Use Which

pandas and PySpark both give Python users DataFrames, and their code can look alike. Underneath they are built for different jobs. pandas runs in one process on one machine and holds the whole table in memory. PySpark plans work lazily and spreads it across many machines. The question is not which is better, but which fits the size of the data, the latency you need and where the job runs.

The same question in both

To compare fairly, here is one query, trips and revenue by pickup hour, written in both libraries on January 2024 New York yellow taxi trips (NYC TLC: trip record data), with a check that the answers agree:

import os
import urllib.request
from pathlib import Path
import pandas as pd
from pyspark.sql import SparkSession, functions as F

path = "yellow_tripdata_2024-01.parquet"
if not Path(path).exists():
    urllib.request.urlretrieve(f"https://d37ci6vzurychx.cloudfront.net/trip-data/{path}", path)

# pandas: eager, everything in this process's memory
df = pd.read_parquet(path)
print(f"Parquet file on disk:  {os.path.getsize(path) / 1e6:6.1f} MB")
print(f"pandas DataFrame in memory: {df.memory_usage(deep=True).sum() / 1e6:6.1f} MB for {len(df)} rows")
pdf_result = (df[df["fare_amount"] > 0]
              .assign(hour=lambda d: d["tpep_pickup_datetime"].dt.hour)
              .groupby("hour")
              .agg(trips=("fare_amount", "size"), revenue=("total_amount", "sum"))
              .reset_index())

# PySpark: lazy, the same question
spark = SparkSession.builder.master("local[4]").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
spark_result = (spark.read.parquet(path)
                .where(F.col("fare_amount") > 0)
                .groupBy(F.hour("tpep_pickup_datetime").alias("hour"))
                .agg(F.count("*").alias("trips"), F.sum("total_amount").alias("revenue"))
                .orderBy("hour")
                .toPandas())
spark.stop()

same_trips = (pdf_result["trips"].to_numpy() == spark_result["trips"].to_numpy()).all()
max_gap = (pdf_result["revenue"] - spark_result["revenue"]).abs().max()
print("same trip counts in all 24 hours:", same_trips)
print(f"largest revenue difference between the two: {max_gap:.6f}")
print(pdf_result.nlargest(3, "trips").to_string(index=False))

Parquet file on disk:    50.0 MB
pandas DataFrame in memory:  418.0 MB for 2964624 rows
same trip counts in all 24 hours: True
largest revenue difference between the two: 0.000000
 hour  trips    revenue
   18 210057 5658931.89
   17 203768 5757616.01
   16 187826 5611107.39

Both give the same answer, which is the point: the choice is about how the work runs, not what it computes. The evening rush from 4 pm to 7 pm is the busiest period in either tool.

Memory: the number that decides most cases

The 50 MB Parquet file became 418 MB in pandas, more than eight times larger. Parquet is compressed and encoded on disk; in memory, pandas holds every value uncompressed. pandas' own documentation is candid about the consequence: it provides data structures for in-memory analytics, so data larger than memory is tricky, and even data that is a sizable fraction of memory becomes unwieldy because some operations make intermediate copies (pandas docs: scaling to large datasets).

At that ratio, a year of this data, twelve files like this one, would need around 5 GB in pandas before any intermediate copies: risky on a 16 GB laptop, while the same pandas code on a single month is comfortable. The pandas guide lists the first things to try before switching tools: load only the columns you need, use efficient types such as categoricals for repeated text, and process in chunks.

Eager versus lazy

pandas executes each line immediately, which makes it pleasant to explore interactively: every step returns a result you can inspect. PySpark builds a plan and runs it at an action, which lets it optimise the whole query, reading only the needed columns and pushing filters into the scan, and split work across executors. The cost is overhead: starting a Spark session and scheduling jobs takes seconds, so on small data pandas usually finishes first. Apache Spark explained for beginners shows the lazy plan in action.

API differences that trip people up

The absence of a row index in Spark is the biggest mental shift: there is no "row 5" in a distributed table unless you create an ordering.

Moving between them

Real pipelines often use both: Spark to reduce billions of rows to a summary, pandas to analyse or chart the summary. toPandas() converts a Spark DataFrame, as in the example, but it collects every record to the driver, so the PySpark documentation advises using it only on a small subset (PySpark docs: Arrow). Setting spark.sql.execution.arrow.pyspark.enabled to true, off by default, makes these conversions faster, and pandas UDFs use Arrow to run vectorised pandas code inside Spark.

There is also the pandas API on Spark, imported as pyspark.pandas, which offers pandas-style syntax running on Spark (PySpark docs: pandas API on Spark). It eases migration of existing code, but its own best practices, such as avoiding shuffles and choosing a distributed default index, show that distributed behaviour still has to be understood.

The middle ground: single-machine engines

Between pandas and a Spark cluster sit engines such as Polars and DuckDB. Both run on one machine but are built for analytical workloads: Polars has a lazy API that optimises a whole query before running it, much as Spark does, and DuckDB runs SQL directly over Parquet files without loading them into a DataFrame first. For data from a few gigabytes up to what one large machine can process, they often remove the need for a cluster. Spark remains the choice when data or throughput outgrows a single machine, or when the rest of the platform is already Spark; when Spark jobs themselves are slow, Spark performance tuning is the next step.

A decision guide

• Data fits comfortably in memory, with room for copies: pandas, or a faster single-machine engine such as Polars or DuckDB.
• Data is larger than one machine's memory, or growing fast: PySpark.
• The team's pipelines already run on Databricks, EMR or Dataproc: PySpark, for consistency, even for medium-sized jobs.
• Interactive exploration and charts on a summary: pandas.
• Scheduled transformations over a data lake: PySpark.

A sensible habit: prototype on a sample in pandas, then write the production job in PySpark when size demands it. The PySpark tutorial covers the DataFrame API you would move to.

Data processing with pandas and the PySpark DataFrame API are both covered in the Data Engineering course, in its Python and Apache Spark modules.

Sources

• Scaling to large datasets (pandas documentation) — pandas
• Apache Arrow in PySpark (PySpark documentation) — Apache Spark
• Pandas API on Spark (PySpark documentation) — Apache Spark
• TLC Trip Record Data (NYC Taxi and Limousine Commission) — NYC Taxi and Limousine Commission