Bezen

Pandas · Module 8 of 22 · 5 concepts · about 6 min

Handling Missing Data

Absence is data. What you do about it changes the answer, so do it deliberately.

At work: Real data always has gaps: a sensor that dropped out, a form field left blank, a price feed that paused.

NaN, None and NaT

What it is

Three different absences: NaN is a float, NaT is its datetime equivalent, None is Python's null. Pandas mostly normalises them, but their types differ.

Why it matters

NaN is not equal to anything, including itself, so df[df.col == np.nan] silently returns nothing. .isna() is the only reliable test.

In the real world: A customer with no cancellation date hasn't cancelled. Treating that blank as “missing data” rather than “still active” would wreck a churn report.

np.nan == np.nan          # False — this is why == never works
df["col"].isna().sum()    # the only reliable count
pd.Series([1, None]).dtype  # float64: None became NaN and upcast the column

NaN being a float means one missing value upcasts an integer column to float64. Ids become 1.0, and joining them against a clean integer column matches nothing.

isna, notna and counting gaps

What it is

isna() gives a boolean frame; df.isna().sum() counts missing values per column; df.isna().mean() gives the proportion.

Why it matters

The proportion is the number that decides the strategy — a column that is 2% missing and one that is 80% missing are different problems.

In the real world: Checking how many customer records lack an email before a marketing campaign; if 40% are missing, the campaign plan changes.

df.isna().mean().sort_values(ascending=False).head(10)
# address_2    0.91   <- drop the column, not the rows
# phone        0.04   <- impute or leave

Practise it: Fill Missing Prices With the Median

dropna

What it is

Drops rows (or columns, with axis=1) containing missing values, with subset to limit which columns count and how to require all rather than any.

Why it matters

Sometimes correct, particularly for a target variable you cannot invent.

In the real world: Removing survey responses that skipped the satisfaction question before averaging satisfaction scores.

Visual example

Missing data

dropna removes whole rows. subset limits which columns can trigger it.

df
useragenote
a31NaN
bNaNvip
c28NaN
dropna(subset=["age"])
2 rows left
useragenote
a31NaN
c28NaN

What each call keeps

dropna(subset=["age"]) → a, c

dropna() → nothing

Why this matters

Missing data is rarely spread evenly. A bare dropna() can quietly remove most of a table because of one column nobody uses.

Common misconception

Wrong idea: dropna() removes the missing values.

Correct idea: it removes every row with a NaN anywhere — here all three — unless you pass subset.

df.dropna(subset=["target"])        # only where the label is missing
df.dropna(how="all")                # rows that are entirely empty
df.dropna(axis=1, thresh=len(df) * 0.5)  # drop columns >50% missing

Dropping rows biases the dataset whenever missingness correlates with the outcome — which it usually does. A default dropna() across all columns can also remove most of the data because of one sparse column you did not care about.

fillna, ffill and bfill

What it is

fillna(value) substitutes a constant or a per-column dict; ffill carries the last valid value forward, bfill carries the next one back.

Why it matters

Forward fill is the right default for time series where a value persists until it changes — a price, a status, a configuration.

In the real world: A stock price feed misses a few minutes; carrying the last known price forward keeps a trading dashboard showing a sensible value.

Visual example

Missing data

Forward fill carries the last known value into a gap — using only what was known at that moment.

price, with gaps
price
09:00100
09:01NaN
09:02NaN
09:03104
ffill()
carried forward
price
09:00100
09:01100
09:02100
09:03104

Where each filled value came from

09:01 ← 100 (09:00)

09:02 ← 100 (09:00)

bfill: 09:01 ← 104 ✗

Why this matters

In a time series the fill direction decides whether a feature can see the future. ffill cannot; bfill does.

Common misconception

Wrong idea: bfill is just the other direction, and just as safe.

Correct idea: bfill copies a future value backwards. A model trained on it sees prices that did not exist yet.

  1. 1Find the gap09:01 and 09:02 are NaN.
  2. 2Look backThe last known price is 100, at 09:00.
  3. 3Carry it forwardFill until the next real value, 104.
df.fillna({"country": "unknown", "discount": 0})
prices.sort_index().ffill()          # last known price persists
prices.ffill(limit=3)                # but not indefinitely

Backward fill on a time series pulls a future value into the past. It looks symmetrical with ffill and it is a leak.

Practise it: Fill Missing Prices With the Median

Imputation and where it is fitted

What it is

Filling with a statistic — mean, median, group median — is imputation, and the statistic has to come from somewhere.

Why it matters

Median is more robust than mean on skewed data, and a per-group median is usually better than a global one.

In the real world: Filling missing incomes in a loan-approval model with the median from training data only, so the model isn't secretly shown test data.

medians = train.groupby("city")["income"].median()      # fitted on train only
train["income"] = train["income"].fillna(train["city"].map(medians))
test["income"] = test["income"].fillna(test["city"].map(medians))

Computing the fill value over the whole dataset and then splitting into train and test. The test set's statistics have leaked into training; fit on the training split alone.

Practise it: Fill Missing Prices With the Median