Deep Learning · Module 7 of 26 · 8 concepts · about 17 min

Backpropagation

How one loss efficiently produces gradients for every earlier parameter — and the two bugs that mechanism creates.

At work: The engine that lets a network learn from its mistakes, and the source of many “training isn't working” bugs.

Backpropagation and the chain rule

What it is

Backpropagation applies the chain rule from the scalar loss backward through the computation graph. Each operation receives the gradient of the loss with respect to its output, combines it with its own local derivative, and returns gradients for its inputs and parameters. The forward pass stores the intermediate values required by those local derivatives; the backward pass then visits operations in reverse dependency order. In PyTorch, loss.backward() performs this traversal and accumulates the resulting tensors into every parameter's .grad field.

Why it matters

It makes training networks with millions of parameters feasible: derivatives are reused locally instead of recomputing one full derivative per parameter. It also gives a useful debugging boundary: if loss is finite but gradients are missing, zero, NaN, or attached to unexpected parameters, the computation graph or objective is wrong before the optimiser is ever involved.

In the real world: How a speech model learns which parts of itself caused a mis-heard word.

Visual example

Deep learning

The forward computation runs left to right; gradient information runs from loss back to every parameter.

Backward pass

input → layersstore operations needed for derivatives
prediction → lossone scalar error
loss.backward()chain rule walks graph in reverse
parameter.gradone gradient tensor per learned weight

Backpropagation reuses each local derivative instead of differentiating the entire model separately for every weight.

optimizer.zero_grad()  # gradients accumulate by default
loss = loss_fn(model(x), y)
loss.backward()        # fills parameter.grad
optimizer.step()       # applies the update

Forgetting zero_grad(). PyTorch accumulates gradients intentionally for gradient accumulation; without a deliberate accumulation plan, stale gradients silently change the effective update.

Practise it: ReLU, Forward and Backward, Linear Layer Forward and Backward, Layer Norm, Backwards

The chain rule, worked by hand

What it is

Take the smallest network with two weights: a = w₁x, h = relu(a), y = w₂h, L = (y − t)². The chain rule says the derivative of L with respect to any earlier quantity is the product of the local derivatives along the path from L to it. With x = 2, w₁ = 1.5, w₂ = 2 and t = 5, the forward pass gives a = 3, h = 3, y = 6, L = 1 — and the backward pass multiplies four small numbers to reach each weight.

Why it matters

Doing it once by hand makes the rest of the module readable. Every idea below is a property of this product: gradients vanish because the factors are below one, explode because they are above one, are zero because one factor is zero (a dead ReLU contributes exactly 0), and can be blocked entirely by detaching one link. Autograd is doing precisely this arithmetic, at scale, with tensors instead of scalars.

In the real world: Knowing the mechanics makes it much faster to debug a model that isn't learning.

Visual example

Chain rule

One path from the loss back to each weight. Every gradient is a product of the local derivatives along it.

L = (w₂·relu(w₁x) − t)² with x = 2, w₁ = 1.5, w₂ = 2, t = 5

	local derivative	× upstream	= value
∂L/∂y	2(y − t) = 2	—	2
∂L/∂w₂	∂y/∂w₂ = h = 3	× 2	6
∂L/∂h	∂y/∂h = w₂ = 2	× 2	4
∂L/∂a	relu′(3) = 1	× 4	4
∂L/∂w₁	∂a/∂w₁ = x = 2	× 4	8

Forward first, then backward

forward: a = 1.5 × 2 = 3 → h = 3 → y = 2 × 3 = 6 → L = (6 − 5)² = 1

backward: ∂L/∂w₁ = 2(y − t) · w₂ · relu′(a) · x = 2 · 2 · 1 · 2 = 8

note: h = 3 had to be stored during the forward pass to compute ∂L/∂w₂

Why this matters

The highlighted rows are the two numbers the optimiser uses. Everything between them is intermediate bookkeeping that autograd does and then discards.

import torch

x, t = torch.tensor(2.0), torch.tensor(5.0)
w1 = torch.tensor(1.5, requires_grad=True)
w2 = torch.tensor(2.0, requires_grad=True)

a = w1 * x            # 3.0
h = torch.relu(a)     # 3.0
y = w2 * h            # 6.0
loss = (y - t) ** 2   # 1.0
loss.backward()

print(w2.grad, w1.grad)   # tensor(6.) tensor(8.)

Assuming a gradient's size tells you how wrong that weight is. w₁.grad = 8 is larger than w₂.grad = 6 only because x = 2 scaled it; gradient magnitude mixes the error with the activations feeding that weight, which is exactly why unnormalised inputs produce badly balanced updates.

Practise it: Linear Layer Forward and Backward, Layer Norm, Backwards

The computational graph autograd builds

What it is

Every operation on a tensor that requires grad creates a node recording which function produced it and which tensors it consumed. y.grad_fn names that function — AddBackward0, MulBackward0, ReluBackward0 — and y.grad_fn.next_functions points at the nodes that produced its inputs. The graph is built dynamically, as the Python code runs, and is freed after backward() unless you ask to keep it. Leaves are the tensors nobody produced: your parameters and your input data.

Why it matters

backward() needs the graph because it needs to know what to differentiate and in what order. That explains the errors people meet. element 0 of tensors does not require grad means the graph never reached a leaf that wanted a gradient. Trying to backward through the graph a second time means the graph was already freed. And anything that leaves PyTorch — a .item(), a NumPy round-trip, an in-place write to a tensor autograd needed — breaks the chain, so the gradient silently stops at that point.

In the real world: PyTorch records every step, so engineers don't have to hand-derive gradients for a custom model.

import torch

x = torch.tensor([2.0], requires_grad=True)
h = (x * 3).relu()
loss = (h ** 2).sum()

print(loss.grad_fn)                       # <SumBackward0 object ...>
print(loss.grad_fn.next_functions[0][0])  # <PowBackward0 object ...>
print(x.is_leaf, x.grad_fn)               # True None — leaves have no producer

broken = torch.tensor([float((x * 3).item())], requires_grad=True)
print(broken.grad_fn)                     # None — .item() left the graph

Converting to NumPy or a Python float mid-model to 'make the maths clearer'. float(t), t.item() and t.numpy() all exit the graph, so everything upstream of that point receives no gradient — and nothing raises, because the downstream graph is still perfectly valid.

Two phases: record forward, walk backward

What it is

Training is a strict two-phase cycle per batch. The forward phase computes the output and, as a side effect, stores both the graph structure and the intermediate tensors the local derivatives will need — relu keeps the sign of its input, a matmul keeps its operands. The backward phase starts from the scalar loss with a seed gradient of 1, visits each node in reverse topological order, and at each one multiplies the incoming gradient by that node's local derivative. Memory peaks at the end of the forward phase and is released as the backward phase consumes it.

Why it matters

It explains the memory profile of training, which is the constraint that decides your batch size. Activations stored for the backward pass are usually the largest single consumer of GPU memory in a deep model — much more than the weights — which is why torch.no_grad() lets you evaluate at a far larger batch size, and why gradient checkpointing (recompute activations instead of storing them) is the standard trade of compute for memory.

In the real world: Explains why training uses far more memory than serving, which drives GPU sizing and cost.

Visual example

Deep learning

The forward phase builds; the backward phase consumes. Memory peaks between them.

One batch, two phases

forward: layers runactivations + graph stored
loss computedone scalar, seed gradient = 1
backward: reverse ordereach node multiplies by its local derivative
grads written, graph freedmemory returns to the weights alone

torch.no_grad() skips the storing, which is why inference fits a much larger batch than training.

import torch, torch.nn as nn

model = nn.Sequential(nn.Linear(1024, 1024), nn.ReLU(), nn.Linear(1024, 10))
x = torch.randn(256, 1024)

logits = model(x)          # phase 1: graph + activations now held in memory
loss = logits.sum()
loss.backward()            # phase 2: consumed in reverse, then freed
print(model[0].weight.grad.shape)   # torch.Size([1024, 1024])

with torch.no_grad():      # phase 1 only: nothing stored, nothing to free
    model(x)

Modifying an activation in place between the two phases. An in-place op on a tensor the backward pass still needs raises one of the variables needed for gradient computation has been modified; nn.ReLU(inplace=True) is safe because autograd knows about it, an ad-hoc h[h < 0] = 0 is not.

Gradients accumulate: why zero_grad() is not optional

What it is

backward() adds into .grad rather than overwriting it. Without optimizer.zero_grad() before each backward pass, step two uses the sum of batches one and two, step three the sum of one, two and three, and so on. The effective gradient — and therefore the effective learning rate — grows without bound across the epoch. The accumulation is deliberate: it is what makes gradient accumulation across micro-batches possible, and what lets two losses contribute to one update.

Why it matters

This is one of the most common real bugs in deep learning code, and its symptom is misleading. The first few steps look normal, then the loss becomes erratic and eventually diverges — which reads exactly like a learning rate that is too high, so people lower the rate and half-fix it. Knowing that accumulation is the default makes zero_grad() a reflex and converts an hour of confused tuning into a one-line check.

In the real world: Forgetting one line makes a model train wrongly while showing no error at all.

Visual example

The accumulation trap

Three steps on the same batch. Without zero_grad(), the gradient the optimiser sees keeps growing.

model.weight.grad.norm() after each backward()

	with zero_grad()	without	effective step
step 1	0.42	0.42	1× — correct
step 2	0.42	0.84	2× the intended lr
step 3	0.42	1.26	3× the intended lr

Common misconception

Wrong idea: a run that diverges after a few hundred steps has too high a learning rate.

Correct idea: it might. But check zero_grad() first: a missing zero is a learning rate that grows every step, and lowering the rate only delays the divergence.

import torch, torch.nn as nn

model = nn.Linear(4, 1)
x, y = torch.randn(8, 4), torch.randn(8, 1)

for step in range(3):
    loss = nn.MSELoss()(model(x), y)   # the same batch every time
    loss.backward()                     # no zero_grad()
    print(step, model.weight.grad.norm().item())
# the norm grows: 1x, 2x, 3x the same gradient

# Deliberate accumulation over 4 micro-batches, done correctly:
opt.zero_grad()
for micro in micro_batches:
    (nn.MSELoss()(model(micro.x), micro.y) / 4).backward()
opt.step()

Calling zero_grad() after step() instead of before backward(). It usually works — and then breaks the moment someone adds an early continue or an exception handler to the loop, leaving one batch's gradients to leak into the next.

Practise it: ReLU, Forward and Backward

Vanishing and exploding gradients, derived

What it is

A gradient that reaches layer one has passed through every layer above it, multiplying by each one's local derivative on the way. That makes it a product of many factors. If those factors average below one — saturated sigmoids, small initial weights — the product shrinks geometrically and early layers receive essentially nothing. If they average above one, the product grows geometrically and one step wipes out the model. There is no stable middle unless something actively keeps the factors near one.

Why it matters

It is why depth was hard for two decades, and it names the shape of the fix. Everything in module 10 — He and Xavier initialisation, batch and layer normalisation, residual connections, ReLU over sigmoid — is a way of keeping that per-layer factor close to one so the product stays usable across dozens of layers. Residual connections are the clearest case: an additive skip contributes a factor of exactly 1 to the product, giving the gradient a path that cannot decay.

In the real world: Why some deep models never learn, and why the modern designs that fix it matter.

Visual example

Chain rule at depth

Gradient magnitude as it travels back through layers, for three average per-layer factors.

factor ≈ 0.6 (vanishing)factor ≈ 1.0 (healthy)factor ≈ 1.4 (exploding)

Why this matters

Nothing about a single layer is wrong in any of these three cases. The failure is compounding — which is why the fixes in module 10 all target the per-layer factor rather than the layer itself.

import torch, torch.nn as nn

deep = nn.Sequential(*[nn.Sequential(nn.Linear(64, 64), nn.Sigmoid()) for _ in range(12)])
nn.MSELoss()(deep(torch.randn(32, 64)), torch.zeros(32, 64)).backward()

for i in (0, 5, 11):
    print(i, deep[i][0].weight.grad.norm().item())
# layer 0's gradient norm is orders of magnitude below layer 11's

Treating a small gradient norm as 'converged'. Near-zero gradients in the first layers of a deep network usually mean the signal never arrived, not that those weights are already right — compare the norm across layers rather than looking at the global norm alone.

Practise it: ReLU, Forward and Backward

detach(): cutting the graph on purpose

What it is

t.detach() returns a tensor sharing the same storage but with no graph history, so gradients stop there. It is the explicit version of the accidental break that .item() causes. The common uses are all deliberate: computing a target from a model you do not want to train through (a target network in reinforcement learning, a teacher in distillation), stopping gradient flow into a frozen encoder, logging a value without keeping its graph alive, or truncating backpropagation through time in a recurrent model.

Why it matters

Without it, a target computed from a model becomes part of the objective and the model can reduce the loss by moving the target instead of improving the prediction — a self-referential objective that collapses. It is also the standard fix for the memory leak where a running total keeps every batch's graph alive: total += loss.detach() (or loss.item()) holds a number, while total += loss holds an entire computation history.

In the real world: Using a model's output as a fixed target without training through it, common in recommendation and reinforcement setups.

import torch, torch.nn as nn

online, target_net = nn.Linear(4, 1), nn.Linear(4, 1)
x = torch.randn(8, 4)

target = target_net(x).detach()        # a fixed target; no gradient flows into it
loss = nn.MSELoss()(online(x), target)
loss.backward()
print(target.requires_grad, target.grad_fn)   # False None

running = 0.0
running += loss.detach().item()        # a number, not a retained graph

Using detach() to silence element 0 of tensors does not require grad. That error means the graph was already broken; detaching more only moves the break, and the real cause is usually a no_grad() block or a requires_grad=False that is still in scope.

requires_grad and freezing part of a model

What it is

requires_grad is a per-tensor flag that decides whether autograd records operations involving it. Setting it to False on a parameter means no .grad is computed for it and no optimiser will update it, while gradients still flow *through* the layer to anything upstream. This is how a pretrained backbone is frozen: flip the flag on its parameters, and train only the new head. It is a different mechanism from no_grad(), which disables recording for a whole block regardless of any flag.

Why it matters

Freezing is the cheapest form of transfer learning and the usual first experiment on a small dataset: fewer trainable parameters means less overfitting, less memory for optimizer state, and a faster step. The trap is that freezing must be done before the optimiser is constructed — or the optimiser must be given only the parameters you intend to train — because an optimiser holding frozen parameters will still apply weight decay and momentum to them in some configurations.

In the real world: Freezing a pretrained image model and training only the last layer on a few thousand of your own product photos.

import torch, torch.nn as nn

model = nn.Sequential(nn.Linear(64, 128), nn.ReLU(), nn.Linear(128, 10))
for p in model[0].parameters():
    p.requires_grad = False              # freeze the first layer

opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-3)
print(sum(p.numel() for p in model.parameters() if p.requires_grad))   # 1290

nn.CrossEntropyLoss()(model(torch.randn(4, 64)), torch.tensor([0, 1, 2, 3])).backward()
print(model[0].weight.grad, model[2].weight.grad.norm().item())   # None, then a number

Freezing parameters after constructing the optimiser from model.parameters(). The optimiser already holds references to them, and with momentum or weight decay enabled it can keep changing weights whose gradient is None — a frozen layer that quietly drifts.