Bezen

Deep Learning · Module 4 of 26 · 8 concepts · about 16 min

Forward Propagation

How inputs become predictions, one tensor operation at a time — and the two switches that change what the pass does.

At work: What happens each time a model answers a request: a product photo tagged, a transaction scored, a sentence completed.

Forward pass and logits

What it is

The forward pass evaluates the model from inputs to output using the current parameters; it does not learn anything by itself. For classification, the final raw scores are logits: unnormalised evidence for each class, where a larger logit means the model prefers that class relative to the others. Softmax converts a vector of logits into probabilities only when probabilities are needed for interpretation, while an argmax turns them into a hard prediction for reporting. The forward pass must preserve the batch dimension so every input still has a corresponding output and target.

Why it matters

Keeping logits raw gives stable loss calculations and makes it clear which operation belongs to the model versus the loss function. Logit-based losses use numerically stable formulations that avoid overflow when scores are large. It also separates learning from decision policy: a threshold for an alert can change without retraining the model or pretending that a 0.51 probability is the same as high confidence.

In the real world: Each time a customer uploads a photo, one forward pass turns it into scores for each product category.

Visual example

Deep learning

Logits are raw scores; the loss receives them before a hard class decision is made.

Classification forward pass

  1. image batch(32, 3, 224, 224)
  2. modellearned feature transformations
  3. logits(32, 3)one score per class
  4. cross-entropycompares logits with true class IDs

Softmax is for reading probabilities, not for the training input to CrossEntropyLoss.

import torch.nn as nn

logits = model(images)                        # (batch, n_classes)
loss = nn.CrossEntropyLoss()(logits, labels)  # no softmax here
probs = logits.softmax(dim=1)                 # only for display or decisions

Using argmax in the training path. It discards the scores and has no useful gradient; use logits for the loss and reserve argmax for evaluation.

The forward pass as one deterministic function

What it is

Given fixed parameters and a fixed input, the forward pass is a pure function: the same batch produces the same output every time. Nothing about it consults the targets, the loss, or the history of training — it is arithmetic over the weights as they currently stand. That determinism has two documented exceptions: layers with randomness (dropout) or batch-dependent state (BatchNorm) behave differently in training mode, and floating-point reductions on a GPU can differ in the last bits between runs.

Why it matters

It is the property that makes debugging possible. If a forward pass gives a different answer on the same input twice, you have found a real bug — a stray train() mode, an uninitialised buffer, a random augmentation inside the model — and you have found it without involving the optimiser at all. Most 'the model is behaving randomly' reports are this, and they are settled by calling the model twice on one batch and comparing.

In the real world: The same photo should always get the same tag in production, which makes the model testable and auditable.

import torch

model.eval()
x = torch.randn(4, 64)
with torch.no_grad():
    a, b = model(x), model(x)
print(torch.equal(a, b))   # True — if this is False, something in the model is stateful

Expecting determinism across devices or dtypes. Reduction order differs between CPU and GPU kernels and between float32 and bfloat16, so the same model on the same input can differ in the fourth decimal place — compare with torch.allclose, not ==, across hardware.

A shape trace through a three-layer MLP

What it is

Tracing means writing the shape at every boundary: input, after each linear, after each activation, output. For a batch of 32 flattened 28×28 images through 784 → 256 → 64 → 10, the trace is (32, 784) → (32, 256) → (32, 64) → (32, 10), with the activations leaving shapes untouched. The leading 32 never changes; only the trailing feature dimension does.

Why it matters

A written trace turns shape debugging from guessing into reading. It also makes the two ends of the model auditable at a glance: the first number must match what the data loader actually emits after flattening, and the last must match the number of classes the loss expects. Both are things people get wrong once and then spend an hour on.

In the real world: Following data through a model by hand to find why a new feature's input breaks it.

Visual example

Deep learning

Only the trailing dimension moves. The leading 32 is the batch, and no layer touches it.

784 → 256 → 64 → 10, batch of 32

  1. input images(32, 1, 28, 28)
  2. Flatten(32, 784)
  3. Linear(784, 256) + ReLU(32, 256)
  4. Linear(256, 64) + ReLU(32, 64)
  5. Linear(64, 10)(32, 10)logits

Activations preserve shape exactly; every shape change in this diagram comes from a Linear.

import torch, torch.nn as nn

model = nn.Sequential(
    nn.Flatten(),                    # (32, 1, 28, 28) -> (32, 784)
    nn.Linear(784, 256), nn.ReLU(),  # (32, 784)       -> (32, 256)
    nn.Linear(256, 64), nn.ReLU(),   # (32, 256)       -> (32, 64)
    nn.Linear(64, 10),               # (32, 64)        -> (32, 10)
)

x = torch.randn(32, 1, 28, 28)
for layer in model:
    x = layer(x)
    print(type(layer).__name__.ljust(8), tuple(x.shape))

Hard-coding the flattened size. nn.Linear(784, 256) silently assumes 28×28×1 inputs; a change to 32×32 or to three channels makes the first linear wrong, and the error names the matmul rather than the resize that caused it. Compute it with nn.Flatten() on one dummy batch.

train() and eval(): the forward pass has two modes

What it is

model.train() and model.eval() set a boolean flag on every submodule, and two families of layer read it. Dropout zeroes a random fraction of activations in training mode and passes everything through unchanged in eval mode. BatchNorm normalises using the current batch's statistics in training mode — and updates its running averages — while in eval mode it uses those stored running averages and updates nothing. Every other layer ignores the flag entirely.

Why it matters

This is the bug that makes evaluation numbers untrustworthy in both directions. Validating in train mode means dropout is still removing activations, so the score is pessimistic and noisy, and BatchNorm is quietly polluting its running statistics with validation data. Serving a model that was never switched to eval mode gives different predictions for the same input on every call. A model.eval() at the top of every evaluation function is the fix, and it belongs there unconditionally.

In the real world: Forgetting evaluation mode makes a live model give slightly random answers, a hard-to-find production bug.

Visual example

Mode

Two layer families read the mode flag. Everything else is identical in both modes.

Mode-sensitive

(the only two rows that differ)

model.train() vs model.eval()
in train()in eval()
Dropoutzeroes p of activationspasses everything through
BatchNormbatch statistics, updates running averagesrunning averages, updates nothing
LayerNormper-example statisticsper-example statistics — unchanged
Linear / Convweights · input + biasweights · input + bias — unchanged

Common misconception

Wrong idea: model.eval() turns off gradient tracking.

Correct idea: it does not. eval() changes layer behaviour; torch.no_grad() changes gradient recording. Inference wants both, and they are separate calls.

import torch, torch.nn as nn

model = nn.Sequential(nn.Linear(8, 8), nn.Dropout(p=0.5))
x = torch.ones(1, 8)

model.train()
print(torch.equal(model(x), model(x)))   # False — dropout resamples each call

model.eval()
print(torch.equal(model(x), model(x)))   # True — dropout is a no-op

Forgetting to call model.train() again after validating. The next epoch then trains with dropout disabled and BatchNorm frozen on stale statistics — training continues, converges to something worse, and nothing warns you.

Practise it: Batch Norm Behaves Differently at Inference

torch.no_grad() and inference memory

What it is

By default every operation on a tensor that requires grad records itself and keeps the intermediate values the backward pass would need. torch.no_grad() (or torch.inference_mode()) switches that recording off for its block: the forward pass computes the same numbers, stores no graph, and frees each activation as soon as the next layer has consumed it. The output comes back with requires_grad=False.

Why it matters

It is the difference between an inference pass that fits in memory and one that does not. Stored activations, not weights, dominate memory during training — for a large batch through a deep network they are several times the parameter count — so dropping them lets you evaluate at a much larger batch size than you can train at. It is also a correctness guard: a validation loop that accidentally builds a graph can leak memory across iterations by keeping every batch's activations alive through a retained loss tensor.

In the real world: Turning off gradient tracking when serving halves memory use, so each GPU handles more customer requests.

import torch

model.eval()
total, correct = 0, 0
with torch.no_grad():                      # no graph, no stored activations
    for xb, yb in val_loader:
        logits = model(xb)
        correct += (logits.argmax(dim=1) == yb).sum().item()
        total += yb.numel()
print(correct / total)

Accumulating total_loss += loss instead of loss.item() in a loop that does track gradients. Each addition keeps that batch's whole graph alive, so memory grows linearly with the number of batches and the run dies partway through an epoch.

Batching: one matmul, the same weights

What it is

A batched forward pass does not loop over examples. nn.Linear computes X @ W.T + b for the whole (batch, in) matrix at once, and the bias is broadcast across the batch axis exactly as NumPy would broadcast a (out,) vector against a (batch, out) array. The weights are shared: every example in the batch is transformed by the identical matrix, which is why a batch's predictions are independent of one another for a Linear layer.

Why it matters

It is where the hardware speedup comes from — one large matmul saturates a GPU's tensor units, while thousands of small ones spend their time on launch overhead — and it is why batch size is a throughput knob rather than a modelling one for most layers. It also explains the exception worth knowing: BatchNorm breaks the independence, because it normalises using statistics computed across the batch, so one example's prediction genuinely depends on its batch-mates.

In the real world: Grouping incoming requests into one batch lets a single GPU serve thousands of predictions a second.

Visual example

Broadcasting

The same learned matrix is applied to every row. The batch axis passes through; only the feature axis changes.

X — (4, 3)
f₀f₁f₂
ex 0102
ex 1011
ex 2220
ex 3111
layer(X) — one matmul over the whole batch
layer(X) — (4, 2)
h₀h₁
ex 02.1-0.4
ex 11.30.6
ex 21.82.2
ex 31.90.8

What is shared and what is not

weights: one (2, 3) matrix, used for all four rows

bias: one (2,) vector, broadcast down the batch axis

rows: independent — for Linear, but not for BatchNorm

Why this matters

Batching is not an optimisation bolted on afterwards. Every PyTorch layer is written batch-first, which is why a single example still has to be shaped (1, features).

import torch, torch.nn as nn

layer = nn.Linear(4, 3)
X = torch.randn(64, 4)

batched = layer(X)                                   # one (64,4) @ (4,3) matmul
looped = torch.stack([layer(row.unsqueeze(0))[0] for row in X])
print(torch.allclose(batched, looped, atol=1e-6))    # True — identical results
print(batched.shape)                                 # torch.Size([64, 3])

Assuming batch independence holds everywhere. With nn.BatchNorm1d in the model, the same example gets a different prediction depending on which other examples share its batch — which is exactly why eval mode switches to fixed running statistics.

Debugging a shape mismatch: a trace that breaks

What it is

Walk a deliberately broken stack: Flatten → Linear(784, 256) → ReLU → Linear(128, 10). Construction succeeds, because PyTorch never compares one layer's out_features with the next layer's in_features. The first real batch raises mat1 and mat2 shapes cannot be multiplied (32x256 and 128x10): mat1 is the activation that arrived, mat2 is the weight it met. The arriving 256 is correct and the declared 128 is the mistake.

Why it matters

Reading that message correctly is a skill worth ten minutes. The error is raised *inside* the second linear layer, so the instinct is to change the second layer — sometimes right, often wrong. The reliable procedure is to print the shape after each layer until one disagrees with what you expected, then fix whichever end you meant to change. Doing it on a two-example dummy batch takes a second and never touches the GPU.

In the real world: The most common error when a data scientist changes an input's size, like a new image resolution.

Visual example

Debugging

The error names two shapes. The left one is what arrived; the right one is what was declared.

mat1 and mat2 shapes cannot be multiplied (2x256 and 128x10)
expected inproducedagrees?
Flatten(2, 1, 28, 28)(2, 784)yes
Linear(784, 256)784(2, 256)yes
ReLUany(2, 256)yes
Linear(128, 10)128—no: 256 arrived
  1. 1Read both numbers2x256 is the activation; 128x10 is the weight it was multiplied against.
  2. 2Trace until they disagreePrint the shape after each layer on a two-example dummy batch.
  3. 3Fix the declaration, not the dataChange 128 to 256 — reshaping the activation would break the batch alignment.
import torch, torch.nn as nn

broken = nn.Sequential(
    nn.Flatten(),
    nn.Linear(784, 256), nn.ReLU(),
    nn.Linear(128, 10),          # wrong: 256 arrives here, not 128
)

x = torch.randn(2, 1, 28, 28)    # a dummy batch, deliberately tiny
try:
    broken(x)
except RuntimeError as error:
    print(error)
    # mat1 and mat2 shapes cannot be multiplied (2x256 and 128x10)

for layer in broken[:3]:
    x = layer(x)
    print(type(layer).__name__, tuple(x.shape))   # last line prints (2, 256)

Fixing the mismatch by reshaping the activation. x.view(-1, 128) makes the error disappear and silently turns 2 examples of 256 features into 4 rows of 128 — the batch axis no longer lines up with the labels, and the loss is now comparing unrelated things.

Writing forward() yourself

What it is

nn.Sequential only covers models that are a straight chain. Anything with a skip connection, two inputs, a branch, or a shape manipulation between layers needs an nn.Module subclass: assign submodules in __init__ so they are registered as children, and describe the data flow in forward. Call the module as model(x), not model.forward(x), so registered hooks still run.

Why it matters

Registration is the part that bites. A layer stored in a plain Python list is not a child module: it will not appear in model.parameters(), so the optimiser never updates it; it will not move with model.to(device); and it will not be saved in the state_dict. The model trains, badly, with a silently frozen layer in the middle. nn.ModuleList and nn.ModuleDict exist precisely to make a collection of layers visible.

In the real world: Building a custom model for a company's own data, like combining product images with their text descriptions.

import torch, torch.nn as nn

class ResidualMLP(nn.Module):
    def __init__(self, dim=64, hidden=128):
        super().__init__()
        self.up = nn.Linear(dim, hidden)
        self.down = nn.Linear(hidden, dim)
        self.act = nn.GELU()
        self.head = nn.Linear(dim, 10)

    def forward(self, x):
        h = self.down(self.act(self.up(x)))
        return self.head(x + h)          # the skip is why Sequential cannot express this

model = ResidualMLP()
print(model(torch.randn(8, 64)).shape)   # torch.Size([8, 10])
print(len(list(model.parameters())))     # 8 — four layers, weight and bias each

Storing submodules in a Python list — self.layers = [nn.Linear(64, 64) for _ in range(4)]. They are invisible to parameters(), to() and state_dict(), so they stay on the CPU, stay at their random initialisation, and are missing from every checkpoint. Use nn.ModuleList.