Pandas · Module 11 of 22 · 4 concepts · about 4 min

Working with Strings

The .str accessor, and enough regex to pull structure out of text nobody validated.

At work: Getting structure out of text: email domains, product codes, addresses, support tickets and log lines.

The .str accessor

What it is

Applies vectorised string operations over a column: .str.strip(), .str.lower(), .str.len(), .str.contains(), .str.startswith().

Why it matters

It is the vectorised path. The alternative — apply(lambda s: s.lower()) — is a Python-level loop for no benefit.

In the real world: Lower-casing and trimming 2 million email addresses at once so duplicate customers can be found.

df["email"].str.lower().str.strip()
df["note"].str.len()
df[df["sku"].str.startswith("SKU", na=False)]

.str on a column that is not actually strings returns NaN for every non-string element, silently, rather than raising.

Practise it: Pull the Month Out of a Timestamp, Tidy Up Column Names

split and expand

What it is

.str.split(sep) gives lists; .str.split(sep, expand=True) gives a DataFrame of columns; n= limits the number of splits.

Why it matters

Compound fields — "city, country", "2024-05-01T10:00" — arrive in one column constantly.

In the real world: Splitting “Bengaluru, Karnataka” into city and state columns for a delivery-coverage report.

Visual example

Strings

expand=True turns one compound column into real columns you can group and join on.

one compound column

location
Berlin, DE
Lisbon, PT

.str.split(", ", expand=True)

two real columns

city	country
Berlin	DE
Lisbon	PT

Why this matters

A column of lists is an object column: nothing downstream can group by it, join on it or write it to Parquet cleanly.

Common misconception

Wrong idea: .str.split gives you new columns.

Correct idea: on its own it gives one column of Python lists. expand=True is what makes columns.

parts = df["location"].str.split(",", n=1, expand=True)
df["city"] = parts[0].str.strip()
df["country"] = parts[1].str.strip()

Leaving the result as lists in an object column. Nothing downstream can group or join on that; expand it into real columns.

extract and regex

What it is

.str.extract(pattern) pulls capture groups into columns; .str.extractall returns every match; .str.replace(..., regex=True) rewrites.

Why it matters

Identifiers arrive embedded in human-written notes far more often than in their own column, and a capture group is how you get them out without a loop.

In the real world: Pulling an order number like “ORD-48213” out of free-text support tickets to link each complaint to its order.

df["sku"] = df["note"].str.extract(r"\b(SKU\d{4})\b")
df[["year", "month"]] = df["period"].str.extract(r"(\d{4})-(\d{2})")

Matching without word boundaries, so a search for SKU12 also matches SKU123. Anchor the pattern.

Practise it: Tidy Up Column Names, Pull the SKU Out of a Free-Text Note

contains, and the na argument

What it is

.str.contains(pattern, na=False) tests each value, with na deciding what a missing value evaluates to.

Why it matters

Without na=False the mask contains NaN, and indexing with a mask that is not purely boolean raises.

In the real world: Finding every product review that mentions “refund”, without the job crashing on reviews that have no text.

df[df["note"].str.contains("refund", case=False, na=False)]
df["note"].str.contains("a.b", regex=False, na=False)   # literal, not regex