Bezen

Pandas · Module 13 of 22 · 6 concepts · about 7 min

GroupBy: The Core of Data Analysis

Split, apply, combine — and the order those steps must happen in.

At work: Almost every business question is “per something”: revenue per region, churn per plan, orders per customer.

The split-apply-combine model

What it is

groupby(key) splits rows into groups, applies a function to each, and combines the results. The key becomes the index unless you pass as_index=False.

Why it matters

It is the workhorse of feature engineering: per-user counts, per-store averages, per-session durations are all one group-by.

In the real world: Total revenue per region: split orders by region, add up each group, and put the results side by side.

Visual example

GroupBy

groupby splits rows by key, applies a function to each group, then combines one answer per group.

df
usertotal
a10
b7
a5
split
group a
usertotal
a10
a5
group b
usertotal
b7
combine
groupby("user").sum()
total
a15
b7

Apply: sum each group

a: 10 + 5 = 15

b: 7 = 7

Why this matters

Per-user counts, per-store averages, per-session durations — most feature engineering is one group-by.

Common misconception

Wrong idea: groupby gives back the grouped rows.

Correct idea: it gives one row per group, with the key as the index — pass as_index=False to keep it as a column.

  1. 1SplitRows are sorted into one group per key.
  2. 2ApplyThe function runs on each group separately.
  3. 3CombineThe answers are stacked, keyed by group.
df.groupby("customer_id")["total"].sum()
df.groupby(["country", "tier"])["total"].mean()
df.groupby("customer_id", as_index=False)["total"].sum()   # key stays a column

agg, and multiple aggregations

What it is

agg reduces each group to one row. Named aggregation — agg(spend=("total", "sum"), orders=("id", "count")) — produces flat, named output columns.

Why it matters

Named aggregation is the readable form: the alternative produces a MultiIndex on the columns that then needs flattening.

In the real world: One table per product category showing total sales, number of orders and average order value, the heart of a weekly business review.

df.groupby("customer_id").agg(
    spend=("total", "sum"),
    orders=("order_id", "count"),
    first_seen=("ordered_at", "min"),
    distinct_skus=("sku", "nunique"),
)

Passing a list of functions and then fighting the resulting MultiIndex columns for the rest of the pipeline.

transform

What it is

Returns a result the same shape as the input, broadcasting each group's value back onto every row of that group.

Why it matters

It is how you attach a group-level statistic to each row — a per-customer average next to each of that customer's transactions — without a join.

In the real world: Showing each order's share of its customer's total spend, to spot customers who depend on one big purchase.

Visual example

GroupBy

transform computes a group statistic and writes it back onto every row of that group.

df — 3 rows
usertotal
a10
b7
a5
groupby("user").transform("sum")
still 3 rows
usertotaluser_total
a1015
b77
a515

Each group's total, broadcast back

a: 10 + 5 = 15 → both a rows

b: 7 → the b row

Why this matters

It is how you put a group statistic next to each row — share of a user's spend, distance from the group mean — without a join.

Common misconception

Wrong idea: transform is agg with a different name.

Correct idea: agg returns one row per group; transform returns one value per original row.

df["user_total"] = df.groupby("user")["total"].transform("sum")
df["pct_of_user_total"] = df["total"] / df["user_total"]
df["above_user_avg"] = df["total"] > df.groupby("user")["total"].transform("mean")

Practise it: Each Product's Share of Its Category

filter, and selecting whole groups

What it is

groupby(...).filter(fn) keeps or drops entire groups based on a predicate over each group.

Why it matters

It expresses 'only customers with at least five orders' directly, rather than computing counts, joining them back and masking.

In the real world: Keeping only stores with at least 100 orders before comparing them, so a store that opened yesterday doesn't top the rankings.

regulars = df.groupby("customer_id").filter(lambda g: len(g) >= 5)
big = df.groupby("customer_id").filter(lambda g: g["total"].sum() > 1000)

Practise it: Filter Rows and Keep Two Columns, Revenue Per Customer From a Messy CSV

Filter before you aggregate

What it is

Rows that should not count — cancellations, refunds, test accounts — must be removed before aggregating, not corrected afterwards.

Why it matters

Aggregation is lossy. Once a refund's negative amount is summed into a customer's revenue, nothing downstream can separate it out.

In the real world: Removing cancelled and test orders before summing revenue; otherwise the monthly number sent to investors is too high.

# Wrong: the refund is already inside the sum
revenue = df.groupby("customer_id")["total"].sum()

# Right: decide what counts, then aggregate
revenue = (
    df[df["status"] == "completed"]
    .groupby("customer_id")["total"]
    .sum()
)

Aggregating first because it makes the frame smaller, then subtracting the unwanted rows. The arithmetic only works for sums and fails silently for counts, means and maxima.

Practise it: Revenue Per Customer From a Messy CSV

Top-N per group, deterministically

What it is

Sort, then take the head of each group. Ties must be broken by an explicit second key.

Why it matters

A pipeline whose output changes between runs on identical input cannot be tested or debugged.

In the real world: The three best-selling products in each category for a homepage, the same three every time the page is built.

top3 = (
    df.sort_values(["customer_id", "total", "order_id"],
                   ascending=[True, False, True])   # order_id breaks ties
      .groupby("customer_id")
      .head(3)
)

Relying on the default sort being stable across versions or across a changed input order. Name the tie-breaker.

Practise it: The Top Three Per Group, Deterministically