Pandas · Module 6 of 22 · 4 concepts · about 4 min

Filtering Rows with Conditions

Boolean masks, and the operator rules that make them behave.

At work: Answering “which ones?”: which orders were refunded, which users churned, which transactions look like fraud.

Boolean masks

What it is

A comparison over a column produces a boolean Series; passing it to df[...] keeps the True rows.

Why it matters

Masks are vectorised and composable, which is what makes filtering fast and readable rather than a loop with an if.

In the real world: Keeping only the orders that were delivered, to report revenue that was actually earned.

Visual example

Filtering

A boolean mask marks each row True or False; indexing with it keeps only the True rows.

df, with the mask

	order	status	mask
0	1	completed	True
1	2	cancelled	False
2	3	completed	True

df[mask]

filtered

	order	status
0	1	completed
2	3	completed

Why this matters

Every filter in Pandas is this: build a True/False Series, then index with it. Combine masks with & and |, each in brackets.

Common misconception

Wrong idea: filtering renumbers the rows 0, 1, 2…

Correct idea: survivors keep their labels (0 and 2 here). Use .iloc or reset_index(drop=True) when you need positions.

1Build the maskdf["status"] == "completed" gives one True/False per row.
2Index with itdf[mask] keeps the True rows only.
3Labels come alongRows keep their original index labels.

mask = df["status"] == "completed"
mask.sum()        # how many rows will survive — check before you filter
df[mask]

Combining with &, | and ~

What it is

Combine masks with & (and), | (or) and ~ (not). Python's and, or and not operate on whole objects and raise for a Series.

Why it matters

Multi-condition filters are the common case, and these are the only operators that work element-wise.

In the real world: Flagging possible fraud: transactions over ₹50,000 and made abroad, or from a card reported lost.

df[(df["status"] == "completed") & (df["total"] > 100)]
df[~df["status"].isin(["cancelled", "refunded"])]

Precedence. & binds tighter than ==, so df[df.a == 1 & df.b == 2] parses as df.a == (1 & df.b) == 2 and does something unrelated. Parenthesise every condition, every time.

isin and between

What it is

isin(values) tests membership against a collection; between(low, high) covers a range, inclusive on both ends by default.

Why it matters

Filtering to a list of ids is a constant feature-engineering step, and isin does it in one pass rather than a chain of | comparisons.

In the real world: Selecting orders from the five cities in a marketing campaign, placed between the campaign's start and end dates.

df[df["customer_id"].isin(vip_ids)]
df[df["total"].between(10, 100)]                 # inclusive of both ends
df[df["total"].between(10, 100, inclusive="left")]  # [10, 100)

between being inclusive at both ends surprises people building non-overlapping buckets, producing rows that land in two buckets at once.

query

What it is

df.query("status == 'completed' and total > 100") expresses a filter as a string, referencing local variables with @name.

Why it matters

It stays readable when a filter has several clauses, and it avoids the parenthesisation trap entirely.

In the real world: Writing a readable filter like “plan == 'premium' and months_active > 12” to find loyal premium customers for a thank-you offer.

floor = 100
df.query("status == 'completed' and total > @floor")
df.query("`order total` > 5")   # backticks for names with spaces

Building a query string from user input or f-strings with unvalidated values. It is evaluated, so it is an injection surface, and a stray quote is a runtime error rather than a wrong answer.