NumPy · Module 13 of 13 · 10 concepts · about 14 min

NumPy for Machine Learning

The array-shaped version of real pipeline problems: holes in the data, window statistics, and the conventions that connect NumPy to a model.

At work: Preparing real, gappy data for models: filling holes, smoothing signals, and handing clean arrays to scikit-learn or PyTorch.

nan as the missing marker

What it is

NumPy has no separate missing value: nan is a float, so only float arrays can carry missingness. Integer and boolean arrays cannot, and casting a column with holes to int destroys them.

Why it matters

It sets the dtype of every feature matrix that has gaps, and it explains why a missing count has to be stored as a float or paired with a mask.

In the real world: Marking missing sensor readings as nan so they can be found and filled before the data goes to a predictive-maintenance model.

X = np.array([[1.0, np.nan], [3.0, 4.0]])
np.isnan(X)                 # the mask, same shape
np.isnan(X).sum(axis=0)     # missing per column: array([0, 1])
np.isnan(X).any(axis=1)     # which rows are incomplete
X[np.isnan(X)] = 0.0        # fill in place, if that is the rule you want

Using a sentinel like -1 or -999 instead. Every reduction then includes it silently — the mean of a column with a thousand -999s is not a mean of anything — and no isnan check will ever find it.

Counting what is present, before trusting it

What it is

Pair every nan-aware statistic with a count: present = (~np.isnan(X)).sum(axis=0). A nan-aware mean over two present values is a number, not an estimate.

Why it matters

It is the difference between a missing-data strategy and a hidden one. A column that is entirely missing needs a decision — drop it, or fill it with a constant — and the count is what surfaces the case.

In the real world: A “90% average uptime” means little if only 2 of 30 days were actually recorded; counting the real readings first avoids a false report.

present = (~np.isnan(X)).sum(axis=0)
mean = np.divide(np.nansum(X, axis=0), present,
                 out=np.zeros(X.shape[1]), where=present > 0)
usable = present >= 10       # columns with enough data to standardise

Calling np.nanmean on an all-nan column. It returns nan with a warning, that nan becomes the fill value for the column, and the column arrives at the model full of nans that came from the imputation step itself.

Standardising columns that have holes

What it is

Compute the per-column mean and standard deviation over the present values only, fill the gaps with that mean, then centre and divide — with a guard for a zero or missing standard deviation.

Why it matters

Real feature matrices have gaps, constant columns, and columns entirely missing for a whole segment. All three turn a naive standardiser into a matrix of nan, and one nan in a batch makes every gradient from it nan.

In the real world: Scaling customer features like income and age for a credit model, when some customers didn't give their income.

def standardize(x):
    x = np.asarray(x, dtype=float)
    present = ~np.isnan(x)
    n = present.sum(axis=0)
    mean = np.divide(np.nansum(x, axis=0), n, out=np.zeros(x.shape[1]), where=n > 0)
    filled = np.where(present, x, mean)
    sd = filled.std(axis=0)
    return np.divide(filled - mean, sd, out=np.zeros_like(filled), where=sd > 0)

Computing the mean and standard deviation over the whole dataset before splitting it. The test set's values then influence the training features — leakage that inflates every offline metric and cannot be reproduced in production, where the future rows do not exist yet. Fit on train, apply to both.

Forward fill with maximum.accumulate

What it is

Build an index array that holds each position's own index where the value is present and -1 (or 0) where it is missing, then take a running maximum: np.maximum.accumulate propagates the last present index forward. Gather with that index array.

Why it matters

It is the vectorised forward fill, over a whole matrix at once, and the same running-maximum idea covers 'last known value', 'most recent event' and 'current session start'.

In the real world: Carrying each machine's last known temperature forward through short gaps in a factory sensor feed.

Visual example

Forward fill

A forward fill is a running maximum over indices: each position learns the last column that had a value.

where(present, col, -1)

	0	1	2	3
x	1	NaN	NaN	4
idx	0	-1	-1	3

maximum.accumulate

accumulated, then gathered

	0	1	2	3
last	0	0	0	3
filled	1	1	1	4

The running maximum

idx: 0, -1, -1, 3

cummax: 0, 0, 0, 3

x[cummax] → 1, 1, 1, 4

Why this matters

It turns a rule stated per row — 'carry the last value forward' — into three whole-array operations, so a matrix with a hundred thousand rows costs one pass instead of a Python loop.

Common misconception

Wrong idea: filling gaps needs a loop, because each cell depends on the one before it.

Correct idea: the dependency is a cumulative operation, and NumPy has those in compiled form. Accumulate the indices, then gather once.

1MarkEach present cell holds its own column number; each gap holds -1.
2Accumulatenp.maximum.accumulate carries the last present index forward.
3Clip and gatherClip -1 to 0 for the leading gap, then take_along_axis.

x = np.array([[1.0, np.nan, np.nan, 4.0]])
present = ~np.isnan(x)
cols = np.broadcast_to(np.arange(x.shape[1]), x.shape)
last = np.maximum.accumulate(np.where(present, cols, -1), axis=1)
last = np.clip(last, 0, None)
filled = np.take_along_axis(x, last, axis=1)
filled    # array([[1., 1., 1., 4.]])

Forgetting the leading gap. Before the first present value the running maximum is still -1, and -1 gathers the *last* column of the row — a value from the future, silently. Clip the index array before gathering.

Practise it: Fill the Gaps in Every Row

Backward fill by reversing

What it is

A backward fill is a forward fill on the reversed array: np.maximum.accumulate(rev, axis=1) on x[:, ::-1], then reverse the result. Reversal is a view, so it costs nothing.

Why it matters

Interior interpolation needs both neighbours — the nearest present value before and after each gap — and the 'after' index is exactly a backward fill.

In the real world: Filling the first missing hours of a new store's data with its first real reading, so the forecast has a starting point.

rev = np.where(present, cols, x.shape[1])[:, ::-1]
nxt = np.minimum.accumulate(rev, axis=1)[:, ::-1]
nxt = np.clip(nxt, None, x.shape[1] - 1)   # trailing gap: no value after it

Initialising the backward pass with -1, copied from the forward one. It needs a sentinel *larger* than any index and a running minimum, otherwise every trailing gap gathers position 0.

Linear interpolation

What it is

np.interp(x, xp, fp) interpolates 1-D data at new positions, holding the end values outside the range. For a whole matrix, compute the previous and next known index arrays and interpolate arithmetically between them.

Why it matters

Interpolation is the right fill for a sensor series where a gap means 'not sampled' rather than 'zero', and the array formulation is what makes it affordable per row.

In the real world: Estimating a delivery truck's position between two GPS pings for live tracking.

xp = np.array([0.0, 3.0])
fp = np.array([1.0, 4.0])
np.interp([0.0, 1.0, 2.0, 3.0], xp, fp)    # array([1., 2., 3., 4.])

# Matrix form, once prev and nxt index arrays exist:
span = np.maximum(nxt - prev, 1)
w = (cols - prev) / span
filled = np.take_along_axis(x, prev, axis=1) * (1 - w) + np.take_along_axis(x, nxt, axis=1) * w

Interpolating a leading or trailing gap. There is only one neighbour, so the arithmetic divides by a zero span; the rule there is to hold the nearest known value, which falls out of the same expression once prev and nxt are clipped to the same index.

Practise it: Fill the Gaps in Every Row

Moving average from prefix sums

What it is

A window sum is a difference of two running totals: prepend a zero to cumsum, then subtract the shifted copy. (c[w:] - c[:-w]) / w gives every window mean in two passes.

Why it matters

It is O(n) regardless of window width, where the naive version is O(n * w). On a 200,000-point series with a 5,000-point window that is the difference between instant and a second per call.

In the real world: A 7-day average of daily active users across years of data, computed in one fast pass.

Visual example

Prefix sums

A window sum is a difference of two prefix sums — the cost does not grow with the window width.

(cumsum[w:] - cumsum[:-w]) / w

input

output

Two passes, not w per position

cumsum: 0 1 3 6 10 15

(6 - 0) / 3 = 2

(10 - 1) / 3 = 3

(15 - 3) / 3 = 4

Why this matters

Every smoothed training curve, latency dashboard and time-series feature is a moving average, usually over a window wide enough that the naive version dominates the runtime.

Common misconception

Wrong idea: the output has the same length as the input.

Correct idea: there are only n - w + 1 full windows. Padding to length n is a separate decision about what the incomplete windows at the start should hold.

1Accumulatecumsum with a 0 prepended, so 'the sum of nothing' has a slot.
2Subtract shiftedc[w:] - c[:-w] is the sum of each window, in one vectorised step.
3DivideBy the window width, for the mean.

def sliding_window_mean(x, w):
    x = np.asarray(x, dtype=float)
    c = np.concatenate(([0.0], np.cumsum(x)))
    return (c[w:] - c[:-w]) / w

sliding_window_mean([1, 2, 3, 4, 5], 3)    # array([2., 3., 4.])

Accumulating in the input's dtype. cumsum on an integer array can overflow, and on float32 a long series loses the precision the subtraction then depends on — the difference of two large nearly-equal sums. Cast to float64 first.

Practise it: Moving Average Without a Loop

diff, cumsum and the running extremes

What it is

np.diff gives consecutive differences (one element shorter), np.cumsum/np.cumprod the running totals, and np.maximum.accumulate/np.minimum.accumulate the running extremes. All take axis=.

Why it matters

Deltas between timestamps, a running peak for a drawdown, a cumulative count of events — these are the time-series primitives, and each is one call rather than a loop with a state variable.

In the real world: Day-on-day change in account balance, a running total of spend, and the highest balance so far, all features for a fraud model.

Visual example

accumulate

A running maximum never looks forward: each position holds the largest value seen up to and including itself.

np.maximum.accumulate(equity)

input

100

120

130

output

100

120

130

Position by position

max(100) = 100

max(100, 120) = 120

max(…, 90) = 120

max(…, 130) = 130

Why this matters

Anything phrased as 'the best so far' — a peak for drawdown, the last known index for a forward fill — is one accumulate, and it can never leak information from the future.

Common misconception

Wrong idea: accumulate is the same as the reduction, repeated.

Correct idea: a reduction returns one value; accumulate returns the whole running history, the same length as the input.

equity = np.array([100.0, 120.0, 90.0, 130.0])
np.diff(equity)                          # array([ 20., -30.,  40.])
peak = np.maximum.accumulate(equity)     # array([100., 120., 120., 130.])
drawdown = equity / peak - 1.0           # array([ 0., 0., -0.25, 0.])
np.diff(ts).min()                        # the smallest gap between events

Forgetting that diff shortens the array by one, then assigning it back beside the original data. Prepend with np.diff(x, prepend=x[0]) so the lengths still line up.

Practise it: Running Share of a Total

Feature matrix conventions: dtype and shape

What it is

A model wants X as a 2-D array of shape (n_samples, n_features), y as a 1-D array of shape (n_samples,), and both cast to float32 — half the memory of the float64 most NumPy code defaults to, and what most training loops expect regardless.

Why it matters

The shape mismatches that look like model bugs are almost always this: a (n,) array where (n, 1) was needed, or a float64 array silently doubling memory and copied again on the way into a framework that wants float32.

In the real world: Handing a model one row per customer and one column per feature, in 32-bit numbers, is what scikit-learn and PyTorch expect.

X = df[["f1", "f2", "f3"]].to_numpy(dtype=np.float32)   # (n, 3)
y = df["target"].to_numpy(dtype=np.float32)             # (n,), not (n, 1)
X.flags["C_CONTIGUOUS"], X.dtype
# True, float32 — what a loader expects

Leaving a single feature as (n,) when a function documents (n, 1). arr[:, None] adds the axis back; it is not the same shape as arr even though the values are identical.

Practise it: Flatten a Batch of Images, One-Hot Encode a Label Vector, Moving Average Without a Loop

Zero-copy interoperability: NumPy, pandas and tensors

What it is

df.to_numpy(), torch.from_numpy(arr) and tf.constant(arr) can all share the same underlying buffer instead of duplicating it — no copy happens as long as the dtype and memory layout already match what the destination wants.

Why it matters

A pipeline that copies at every boundary — DataFrame to array, array to tensor — pays for it twice: once in time, once in peak memory, on data that was already sitting in a buffer the whole time.

In the real world: Passing a large feature table from pandas to PyTorch without copying it, so training starts faster and uses less memory.

import pandas as pd
arr = df.to_numpy()          # a view when df's dtype is already uniform
t = torch.from_numpy(arr)    # shares memory with arr — writing to t writes to arr
t.numpy() is not arr          # a different NumPy view, same buffer

Mutating a tensor made with torch.from_numpy and being surprised the original array changed too. Shared memory is the point — copy explicitly with .clone() or arr.copy() when independence is what you actually need.

Practise it: Class Weights From Label Counts, Flatten a Batch of Images, One-Hot Encode a Label Vector, Pairwise Euclidean Distances, Pick the Rows That Pass a Threshold, Reproducible Train and Test Indices, Running Share of a Total, Scale Every Column to 0 and 1