Pandas · Module 21 of 22 · 5 concepts · about 5 min

Debugging Common Pandas Problems

The five bugs everyone hits, and what each one is actually telling you.

At work: The bugs that quietly put wrong numbers in a report or a model, and how to catch them before anyone else does.

SettingWithCopyWarning

What it is

Some operations return a view onto the original data and some return a copy. Assigning into the result of a chained selection may write to a temporary that is then discarded.

Why it matters

It is the classic 'my assignment did nothing' bug, and the warning is telling you Pandas could not determine whether your write reached the original frame.

In the real world: An analyst “fixes” prices on a filtered copy and the change never reaches the real table, so the wrong prices go out in the next export.

df[df["score"] > 0]["flag"] = True       # may silently do nothing
df.loc[df["score"] > 0, "flag"] = True   # correct: one indexing step

sub = df[df["score"] > 0].copy()          # or take an explicit copy
sub["flag"] = True

Silencing the warning rather than collapsing the chain into a single .loc. It is almost always right.

Unexpected NaNs after an operation

What it is

NaNs that appear from nowhere are nearly always index alignment: a join that did not match, or a Series assigned to a column whose labels differ.

Why it matters

Knowing to suspect the index first turns a baffling result into a two-minute check.

In the real world: Targets added to a sales table come out blank for half the stores because the store names don't match exactly.

print(left.index.difference(right.index)[:10])
print(df["key"].dtype, other["key"].dtype)
print(df.index.equals(other.index))

A join that returned nothing

What it is

Zero matches against data that visibly lines up is a key problem: mismatched dtypes, whitespace, or case.

Why it matters

Screen output hides all three, so the data looks identical while the comparison disagrees.

In the real world: Orders and customers won't join because one side stores IDs as numbers and the other as text with spaces; the report shows zero customers.

print(repr(left["key"].head().tolist()))    # repr shows ' 123 ' vs '123'
print(left["key"].dtype, right["key"].dtype)
left["key"] = left["key"].astype("string").str.strip()
print(len(set(left["key"]) & set(right["key"])), "keys in common")

"123" and 123 never match. Normalise key dtypes and strip whitespace on both sides before merging.

Row counts that changed unexpectedly

What it is

A count that grew after a join is fan-out; one that shrank is an inner join or a filter that matched less than expected.

Why it matters

Checking len(df) before and after every join and filter catches both immediately, while the cause is still one step away.

In the real world: After a join, 10,000 orders become 13,500, a sign that some customers appear twice, before revenue is overstated.

before = len(orders)
joined = orders.merge(customers, on="customer_id", how="left")
assert len(joined) == before, f"fan-out: {before} -> {len(joined)}"

Results that change between runs

What it is

Non-determinism comes from unstable sorts with ties, sample without a random_state, or iteration over a set or dict.

Why it matters

A pipeline that cannot reproduce its own output cannot be tested, and a diff against yesterday's run tells you nothing.

In the real world: A “random” sample for an A/B test changes every time the script runs, so results can't be reproduced when the finance team asks.

df.sample(100, random_state=0)                        # fixed seed
df.sort_values(["score", "id"], ascending=[False, True])  # explicit tie-break