Pandas · Module 20 of 22 · 5 concepts · about 5 min

Pandas Performance

The difference between a job that finishes and one you kill after an hour.

At work: Keeping a daily job fast and affordable as the data grows from thousands of rows to tens of millions.

Vectorisation over loops

What it is

Operations expressed over whole columns run in compiled code; iterrows and apply execute Python per row.

Why it matters

The gap is routinely one to two orders of magnitude, and it is the difference between a feature job taking seconds and taking an hour.

In the real world: Calculating tax on 10 million orders with one column operation takes a second; a row-by-row loop can take many minutes and holds up the daily report.

# slow: Python once per row
df["margin"] = df.apply(lambda r: r["revenue"] - r["cost"], axis=1)

# fast: one compiled pass
df["margin"] = df["revenue"] - df["cost"]

Assuming apply is vectorised because it contains no visible loop. It is a loop with better manners.

Practise it: Sessionise an Event Stream, The Top Three Per Group, Deterministically

Why iterrows is the worst option

What it is

iterrows yields each row as a Series, which means constructing an object per row and losing dtypes in the process.

Why it matters

If iteration is genuinely required, itertuples is substantially faster and keeps types; most of the time the answer is a vectorised expression or a group-by.

In the real world: A nightly billing script that loops over rows gets slower every month as customers grow, until it no longer finishes before morning.

for row in df.itertuples(index=False):    # faster, keeps dtypes
    process(row.order_id, row.total)

# for _, row in df.iterrows():           # a Series built per row

When apply is acceptable

What it is

apply is reasonable for genuinely row-dependent logic with no vectorised equivalent, and on small frames where clarity beats speed.

Why it matters

Being dogmatic costs readability for no gain on a thousand-row frame. The judgement is about size and whether a vectorised form exists.

In the real world: Parsing a few hundred unusual address formats with a custom function is fine; running that on 50 million rows is not.

# no vectorised equivalent, and the frame is small
lookup["parsed"] = lookup["raw"].apply(custom_parser)

# but this has one:
df["tier"] = np.select([df.spend > 1000], ["high"], default="low")

apply(axis=1) on a large frame. That is the slowest path in Pandas and usually replaceable with np.select, where, or arithmetic between columns.

Memory

What it is

df.info(memory_usage="deep") reports true usage including strings. Downcasting numerics and converting low-cardinality strings to category are the two biggest wins.

Why it matters

Most 'Pandas ran out of memory' problems are an object column of repeated strings that should have been categorical.

In the real world: Shrinking a 12 GB dataset to 3 GB by storing numbers more compactly lets the analysis run on a laptop instead of an expensive server.

df.memory_usage(deep=True).sort_values(ascending=False).head()
df["country"] = df["country"].astype("category")
df["count"] = pd.to_numeric(df["count"], downcast="integer")

Chunking and column pruning

What it is

read_csv(..., chunksize=N) streams a file in pieces; usecols avoids reading columns you will not use at all.

Why it matters

The cheapest way to process a file larger than memory is to never load the parts of it you do not need.

In the real world: Processing a year of clickstream logs a chunk at a time, reading only the three columns needed, so a laptop can handle a file far bigger than its memory.

totals = pd.Series(dtype="float64")
for chunk in pd.read_csv("big.csv", usecols=["user_id", "total"], chunksize=500_000):
    totals = totals.add(chunk.groupby("user_id")["total"].sum(), fill_value=0)