Pandas · Module 12 of 22 · 5 concepts · about 5 min

Creating and Transforming Columns

Deriving new columns from existing ones — the heart of feature engineering, and the easiest place to accidentally loop.

At work: Building the numbers a business runs on (margin, age, spend per visit) and the features a model learns from.

Arithmetic between columns

What it is

Column-wise operations produce new columns: df["margin"] = df["revenue"] - df["cost"]. They align on the index and skip missing values per element.

Why it matters

This is the fast path — compiled, element-wise, no Python per row.

In the real world: Calculating profit per order as revenue minus cost minus shipping, the number every retail business watches.

df["margin"] = df["revenue"] - df["cost"]
df["margin_pct"] = df["margin"] / df["revenue"].replace(0, np.nan)

Dividing by a column that contains zeros gives inf rather than an error. Guard it explicitly or the infinity propagates into every later aggregate.

assign and method chaining

What it is

df.assign(margin=lambda d: d.revenue - d.cost) returns a new frame with the column added, which lets transformations chain without intermediate names.

Why it matters

A chain reads top to bottom as a pipeline and never mutates its input, which makes a transformation reproducible and easy to reorder.

In the real world: Building a monthly finance report as one readable sequence of steps, so a colleague can follow and re-run it.

result = (
    df.query("status == 'completed'")
      .assign(margin=lambda d: d.revenue - d.cost)
      .assign(is_big=lambda d: d.margin > 100)   # can reference the previous one
      .sort_values("margin", ascending=False)
)

Practise it: Add a Line Total Column

map versus apply

What it is

Series.map applies a function or a dict lookup element-wise. apply runs a function over a Series or across a DataFrame's rows or columns.

Why it matters

map with a dict is the idiomatic lookup: replacing codes with labels in one vectorised pass.

In the real world: Mapping plan codes like “P1” to names like “Premium” for a report, instead of running slow custom code on every row.

labels = {"a": "Acquisition", "r": "Retention"}
df["channel"] = df["channel_code"].map(labels)
df["channel"].isna().sum()    # unmapped codes became NaN — check this

map with a dict that is missing a key yields NaN rather than raising, so an incomplete mapping silently blanks rows. Pass a Series and check for gaps, or use .fillna() deliberately.

where and mask

What it is

df.where(cond, other) keeps values where the condition holds and substitutes elsewhere; mask is its inverse.

Why it matters

It is the vectorised if, and it preserves shape, which matters when assigning back to a column.

In the real world: Capping refunds at the original order value so a data-entry error can't show a customer being refunded more than they paid.

df["capped"] = df["total"].where(df["total"] < 1000, 1000)   # cap at 1000
df["total"].mask(df["total"] < 0, 0)                          # floor at 0

np.select for multi-way conditions

What it is

np.select([cond1, cond2], [value1, value2], default=...) picks per row from several conditions in order.

Why it matters

It is the readable, vectorised alternative to a chain of nested np.where calls or an apply with if/elif.

In the real world: Sorting customers into bronze, silver and gold tiers by yearly spend for a loyalty programme.

df["tier"] = np.select(
    [df.spend > 1000, df.spend > 100],
    ["high", "medium"],
    default="low",
)

Forgetting that conditions are evaluated in order and the first match wins. Writing the broadest condition first makes every later one unreachable.