Pandas Performance: Making Your Code 10x Faster

Slow pandas code is almost always slow for one reason: it makes Python do, one row at a time, work that pandas could do for a whole column in compiled code. The fixes are mostly mechanical once you can spot the pattern, and the gains are often far larger than 10x. Below, each fix is timed on the same 200,000-row DataFrame, so you can see the size of the effect rather than take it on faith.

The test data and a tiny timer

import time
import numpy as np
import pandas as pd

rng = np.random.default_rng(0)
n = 200_000
df = pd.DataFrame({
    "customer_id": rng.integers(1, 20_001, n),
    "price": rng.uniform(10, 500, n).round(2),
    "qty": rng.integers(1, 10, n),
    "city": rng.choice(["Hyderabad", "Pune", "Chennai", "Delhi"], n),
})

def best_of(fn, repeat=3):
    times = []
    for _ in range(repeat):
        start = time.perf_counter()
        fn()
        times.append(time.perf_counter() - start)
    return min(times)

Taking the best of several runs filters out noise from the rest of the machine. In a notebook, %timeit does the same job.

1. Replace row loops with column arithmetic

The most common slow pattern is computing something per row with iterrows() or apply(axis=1):

slow = best_of(lambda: [row["price"] * row["qty"] for _, row in df.iterrows()], repeat=1)
medium = best_of(lambda: df.apply(lambda r: r["price"] * r["qty"], axis=1), repeat=1)
fast = best_of(lambda: df["price"] * df["qty"])
print(f"iterrows {slow:.2f}s | apply {medium:.2f}s | vectorised {fast * 1000:.1f} ms")
print(f"vectorised is {slow / fast:,.0f}x faster than iterrows")

iterrows 4.46s | apply 1.24s | vectorised 0.8 ms
vectorised is 5,387x faster than iterrows

iterrows builds a new Series object for every row, and the pandas documentation notes that it does not even preserve column types across a row (pandas docs: iterrows). apply(axis=1) is faster but still calls a Python function 200,000 times. df["price"] * df["qty"] hands both whole columns to compiled code in a single operation. Exact numbers vary by machine and run, but a gap of three orders of magnitude is typical, and it is the reason the pandas performance guide starts with vectorisation (pandas docs: enhancing performance).

If you truly need a loop, for example to call an external API per row, itertuples() is much faster than iterrows().

2. Turn if-else logic into np.select

Row-wise logic like "high, mid or low price band" tempts people back into apply:

def band(p):
    return "high" if p > 300 else "mid" if p > 100 else "low"

per_value = best_of(lambda: df["price"].apply(band))
selected = best_of(lambda: np.select([df["price"] > 300, df["price"] > 100], ["high", "mid"], default="low"))
same = (df["price"].apply(band) == np.select([df["price"] > 300, df["price"] > 100], ["high", "mid"], default="low")).all()
print(f"apply {per_value * 1000:.1f} ms | np.select {selected * 1000:.1f} ms | same result: {same}")

apply 25.3 ms | np.select 3.7 ms | same result: True

np.select takes a list of conditions and a matching list of results, and uses the first condition that is true for each row, with a default for the rest (NumPy docs: select). Order the conditions exactly as you would the if/elif chain. For a single condition, np.where(cond, a, b) is simpler; for numeric bins, pd.cut does the job with labelled intervals.

3. Use built-in groupby aggregations

A lambda inside groupby().apply() runs once per group. With many groups, that is many Python calls:

custom = best_of(lambda: df.groupby("customer_id")["price"].apply(lambda s: s.mean()))
builtin = best_of(lambda: df.groupby("customer_id")["price"].mean())
print(f"{df['customer_id'].nunique():,} groups | lambda {custom * 1000:.0f} ms | built-in {builtin * 1000:.1f} ms")

20,000 groups | lambda 565 ms | built-in 10.9 ms

mean, sum, count, min, max, std, nunique and the other named aggregations run in optimised code for all groups at once. Pass their names to agg, for example agg(total=("price", "sum")), instead of wrapping them in lambdas. With only a handful of groups the gap shrinks, because the per-group cost is paid only a few times.

4. Store repetitive text as category

A text column with few distinct values stores the same strings again and again. The category dtype stores each distinct value once, plus a small integer code per row (pandas docs: categorical data):

as_object = df["city"].memory_usage(deep=True) / 1e6
as_category = df["city"].astype("category").memory_usage(deep=True) / 1e6
print(f"object {as_object:.1f} MB -> category {as_category:.1f} MB")

object 12.7 MB -> category 0.2 MB

Less memory also means faster grouping and filtering on that column. Do not convert columns where nearly every value is unique, such as IDs or free text; the category mapping then saves nothing.

5. Load less in the first place

The fastest data is data you never read. Two read_csv arguments do most of the work (pandas docs: scaling to large datasets):

• usecols=[...] reads only the columns you need. A 40-column export where the analysis uses 6 columns shrinks accordingly.
• dtype={...} sets compact types up front, such as "category" for repetitive text, "int32" where values fit, and "float32" where the precision is enough, instead of letting everything default to 64-bit numbers and Python strings.

For files that do not fit in memory, chunksize= returns an iterator of DataFrames; aggregate each chunk and combine the results. Saving intermediate results as Parquet instead of CSV also keeps the types and loads much faster the next time.

Other habits that add up

• Build DataFrames once. Appending rows inside a loop copies data every time. Collect rows in a Python list and call pd.DataFrame(rows) once, or pd.concat a list of frames once.
• Filter early. Drop the rows and columns you do not need before joins and group-bys, not after.
• Avoid chained indexing such as df[df.qty > 5]["price"] = 0. It may modify a temporary copy instead of df; use df.loc[df.qty > 5, "price"] = 0.
• Watch the dtypes. A numeric column that silently became object, for example because one bad value arrived as text, loses the fast vectorised paths. Check df.dtypes after loading and after each merge.
• Measure before optimising. Profile a slow notebook cell by timing its steps. The slow part is usually one line, and usually a loop.

When pandas itself is the limit

Most slow pandas code is slow because of loops, and the fixes above address that. When the data really is large, a few options change the engine rather than the code:

• PyArrow-backed columns. Since pandas 2.0, read_csv(..., dtype_backend="pyarrow") stores columns in Apache Arrow format, which is often more memory-efficient for strings and supports missing values in every type.
• DataFrame.eval and query evaluate long arithmetic or filter expressions on large frames with fewer temporary arrays, as the pandas performance guide describes.
• Numba, a just-in-time compiler, can speed up numeric functions that genuinely cannot be vectorised, such as custom running calculations; some pandas methods accept engine="numba".
• Other DataFrame libraries. Polars, written in Rust, runs many operations in parallel and has a lazy query mode that optimises the whole pipeline before running it. Moving a large, slow pipeline can pay off; moving a notebook that is slow because of iterrows will not, because the loop moves with it.

Try the five fixes first: they usually turn minutes into seconds without changing tools.

Fix in this order

1. Replace iterrows, itertuples and apply(axis=1) with column expressions.
2. Replace conditional apply with np.where, np.select or pd.cut.
3. Use named aggregations in groupby, not lambdas.
4. Load fewer columns, with explicit and compact dtypes.
5. Only then consider parallelism or other tools; multithreading vs multiprocessing explains why threads rarely help pandas code.

Column arithmetic in pandas is NumPy vectorisation underneath, explained in the NumPy guide; the everyday cleaning and aggregation workflow is in the pandas guide. Data cleaning and analysis with pandas is part of the data handling module of the Python Programming course.

Sources

• Enhancing performance — pandas documentation
• pandas.DataFrame.iterrows — pandas documentation
• Categorical data — pandas documentation
• Scaling to large datasets — pandas documentation
• numpy.select — NumPy documentation