NumPy · Module 6 of 13 · 7 concepts · about 9 min

Aggregation & Statistics

Collapsing an array to an answer, and computing per-group answers with integer indices instead of a DataFrame.

At work: Turning millions of numbers into the few a business decides on: totals, averages, the best, the worst, counts per group.

sum, mean, min, max and std

What it is

Reductions that take axis=, keepdims= and dtype=. Available both as methods (x.sum()) and functions (np.sum(x)); np.std uses a population divisor (ddof=0) by default.

Why it matters

They are the whole of descriptive statistics on an array, and ddof is a real difference: scikit-learn's scaler and np.std agree, while pandas.Series.std defaults to ddof=1 and does not.

In the real world: Average basket size, busiest day and how much daily sales vary: the core numbers in a weekly business review.

X.mean(axis=0)                  # per-feature mean
X.std(axis=0, ddof=0)           # population std, as sklearn uses
X.max(axis=1, keepdims=True)    # per-row max, shape (n, 1)
X.sum(dtype=np.float64)         # accumulate wide even if X is float32

Comparing a NumPy std with a Pandas one and concluding the data changed. The divisor differs: n versus n - 1.

Practise it: Accuracy From Model Scores

argmin, argmax and unravel_index

What it is

argmax returns the *position* of the maximum, flat by default or per-axis with axis=. np.unravel_index(flat, shape) converts a flat position back into coordinates.

Why it matters

Predicted class per row is argmax(axis=1), and that one line is the last step of nearly every classifier's inference path.

In the real world: Finding which store had its best day, and which day that was, from a stores-by-days sales grid.

probs = np.array([[0.1, 0.7, 0.2], [0.6, 0.3, 0.1]])
probs.argmax(axis=1)                       # array([1, 0]) — predicted classes
np.unravel_index(probs.argmax(), probs.shape)   # (0, 1) — the global best cell
probs.max(axis=1)                          # the confidence itself

argmax returns only the first maximum and gives no hint that there was a tie. On a model outputting identical probabilities (an untrained one, or a saturated one) every prediction becomes class 0 and the accuracy looks like the base rate for a reason nobody suspects.

Practise it: Accuracy From Model Scores

nan-aware reductions

What it is

np.nanmean, np.nansum, np.nanstd, np.nanmax and friends ignore nan instead of propagating it. np.nanmean of an all-nan slice is nan with a warning.

Why it matters

One missing value turns an ordinary mean into nan, and that nan then spreads through every downstream feature. The nan-aware version is usually what you meant.

In the real world: Averaging sensor readings when some sensors dropped out, without one missing value turning the whole average into “nan”.

X = np.array([[1.0, np.nan], [3.0, 4.0]])
X.mean(axis=0)        # array([2., nan])
np.nanmean(X, axis=0) # array([2., 4.])
np.isnan(X).sum(axis=0)   # how many are missing per column

Reaching for nanmean without checking the count behind it. A column with two present values out of ten thousand gets a confident mean computed from two numbers, and nothing in the output says so.

bincount for counts and group sums

What it is

np.bincount(ids) counts how many times each integer appears, returning an array indexed by id. With weights= it sums the weights per id instead, and minlength= fixes the output length.

Why it matters

It is a group-by in one compiled pass: per-user totals, per-class counts, a histogram of integer ids. Far faster than np.add.at, and the natural way to compute per-group means as sums / counts.

In the real world: Counting orders per product ID, or totalling revenue per store ID, over millions of orders in one fast call.

Visual example

bincount

bincount(ids, weights=values) is split, sum, combine with no DataFrame: the group id *is* the output position.

values and ids

value	id
1	0
2	0
3	1
4	1

split

id 0

value	id
1	0
2	0

id 1

value	id
3	1
4	1

combine

bincount(ids, weights=values)

	sum
0	3
1	7

One pass, one output slot per id

0: 1.0 + 2.0 = 3.0

1: 3.0 + 4.0 = 7.0

Why this matters

Inside a training loop there is no groupby. Per-user, per-slate and per-token aggregates are integer-indexed arrays, and this is how they are computed.

Common misconception

Wrong idea: grouping needs a sort, or a dictionary keyed by group.

Correct idea: with integer ids the id is the address. One compiled pass adds each value into its slot; empty groups come out as 0.

1Ids as addressesEach row's id names the output position it contributes to.
2Accumulateweights=values sums instead of counting.
3Fix the lengthminlength=n_groups keeps empty trailing groups in the result.

values = np.array([1.0, 2.0, 3.0, 4.0])
ids = np.array([0, 0, 1, 1])
sums = np.bincount(ids, weights=values, minlength=2)   # array([3., 7.])
counts = np.bincount(ids, minlength=2)                 # array([2, 2])
means = np.divide(sums, counts, out=np.zeros_like(sums), where=counts > 0)

Letting the output length come from the data. Without minlength=n_groups, a batch whose largest id is 3 returns a length-4 array even though there are ten groups, and the shape mismatch appears somewhere else entirely.

Practise it: A Confusion Matrix in One Pass, Class Weights From Label Counts, Group Sums Without a GroupBy

np.add.at, and 2-D group sums

What it is

np.add.at(out, idx, values) is unbuffered scatter-add: it works for any dtype, any output shape and multi-axis indices, where bincount handles only 1-D integer ids.

Why it matters

It is the general form, accumulating per-(user, item) pairs, or summing gradient contributions into an embedding table, and the correct alternative to out[idx] += values.

In the real world: Building a store-by-product sales table straight from a list of individual orders.

out = np.zeros((3, 2))
rows = np.array([0, 0, 2])
cols = np.array([1, 1, 0])
np.add.at(out, (rows, cols), [1.0, 2.0, 5.0])
out      # [[0., 3.], [0., 0.], [5., 0.]]

Using np.add.at where bincount would do. It is unbuffered and therefore much slower (often an order of magnitude) so for 1-D integer groups prefer bincount, or flatten a pair of ids into one and bincount that.

Practise it: Group Sums Without a GroupBy

Row-major index arithmetic: a confusion matrix in one pass

What it is

A pair of small integers can be encoded as one: i * n + j is unique for i, j in [0, n). Count the encoded pairs with bincount(minlength=n*n) and reshape(n, n).

Why it matters

It turns any fixed-size cross-tabulation into a single compiled pass, and it is the same arithmetic NumPy itself uses to index a row-major buffer; np.ravel_multi_index does it for you.

In the real world: Counting how often a fraud model's predictions match the true outcome, to report how many frauds it catches and misses.

y_true = np.array([0, 1, 1, 2])
y_pred = np.array([0, 1, 2, 2])
n = 3
flat = y_true * n + y_pred            # or np.ravel_multi_index((y_true, y_pred), (n, n))
cm = np.bincount(flat, minlength=n * n).reshape(n, n)
cm.trace() / cm.sum()                 # accuracy, from the matrix

Reshaping without minlength=n * n. If the last class never appears as a prediction, bincount returns a shorter array and the reshape either raises or (with an unlucky length) succeeds and shifts every count into the wrong cell.

Practise it: A Confusion Matrix in One Pass

np.unique and its extras

What it is

np.unique(x) returns the sorted distinct values. return_counts=True adds their frequencies, return_inverse=True gives an index array that rebuilds x from the uniques, and return_index=True gives the first occurrence of each.

Why it matters

return_inverse is the label encoder: it maps arbitrary values to contiguous integers 0..k-1, which is exactly what bincount and one-hot encoding require.

In the real world: Listing the distinct countries customers come from and how many customers each has.

labels = np.array(['cat', 'dog', 'cat', 'fox'])
vals, codes = np.unique(labels, return_inverse=True)
vals                   # array(['cat', 'dog', 'fox'])
codes                  # array([0, 1, 0, 2]) — usable with bincount
np.bincount(codes)     # array([2, 1, 1])

Fitting np.unique separately on train and test data. The codes are positions in each array's own sorted uniques, so a class missing from one split shifts every code after it, a silent label permutation between training and inference.