Normalization & Regularization

Normalization controls activation scale and can make optimisation much less sensitive to the changing distributions produced by earlier layers. BatchNorm normalizes each feature channel using batch statistics and maintains running statistics for inference, which suits many CNNs; LayerNorm normalizes across features within each example, so its behaviour does not depend on the other examples in a batch and is standard in Transformers. Dropout randomly removes activations during training to discourage co-adaptation, while weight decay continuously penalises large weights. These techniques solve different problems and can coexist, but none repairs bad labels or leakage.

These techniques alter optimisation and generalisation in different ways. Knowing which is doing what prevents 'regularisation' from becoming a bag of rituals. In particular, small or uneven batches can make BatchNorm statistics noisy, while LayerNorm has no running-batch state; the architecture and batch regime should guide the choice.

nn.BatchNorm1d(f) normalizes each of the f features independently, using the mean and variance of that feature *across the batch*: x̂ = (x − μ_feature) / sqrt(σ²_feature + eps). It then applies two learned per-feature parameters, γ (scale) and β (shift), so the layer can undo the normalization if that is what the loss prefers — normalization constrains the optimisation path, it does not constrain what the network can represent. For a 4-D image tensor (N, C, H, W), BatchNorm2d treats each channel as one feature and averages over N × H × W positions, which is why its statistics come from N·H·W numbers rather than N.

It removes a specific obstacle: without it, a change to layer 3's weights shifts the distribution layer 4 receives, so layer 4 spends its updates chasing a moving target. Normalizing between them decouples the layers, which in practice permits noticeably larger learning rates and much less sensitivity to the initialization scale of module 10.

In train() mode BatchNorm uses the current batch's statistics, and updates a running mean and variance as a side effect of the forward pass. In eval() mode it uses those accumulated running statistics instead, so a prediction depends only on the input and not on whichever other examples happened to share its batch. This is not an optimisation: it is required for inference to be deterministic and for a batch of one to work at all. It also means BatchNorm's buffers are part of the model's state — state_dict() includes running_mean and running_var, and a checkpoint saved without them predicts differently.

This is one of the most common real bugs in deep-learning code, and it is silent. Validating without model.eval() scores each example against its batch-mates, which makes the metric depend on batch composition and shuffling; serving without it makes a single request's prediction depend on nothing meaningful at all. It ties directly to the .train()/.eval() boundary from module 9 — BatchNorm is the layer that makes that boundary consequential.

LayerNorm normalizes over the *feature* axis within a single example: for one token's 768-dimensional vector it computes that vector's own mean and variance and standardises it. Nothing about the calculation involves any other example, so there are no running statistics, no train/eval difference in what it computes, and a batch of size 1 behaves exactly like a batch of size 512. BatchNorm normalizes over the batch axis per feature; LayerNorm over the feature axis per example. Those are perpendicular directions through the same tensor.

It is why every Transformer (module 18) uses LayerNorm. Sequence models are trained with variable-length inputs, batch sizes of one during autoregressive generation, and heavy gradient accumulation — all regimes where BatchNorm's batch statistics are either noisy or undefined. LayerNorm also never couples two examples in a batch, so one bad example cannot shift another's activations, and training-time behaviour matches inference-time behaviour exactly.

nn.Dropout(p) zeroes each activation independently with probability p on every training forward pass, so the network cannot rely on any particular unit being present and must spread a feature across several units. That zeroing reduces the expected sum entering the next layer by a factor of (1 − p), which would make training and inference see systematically different magnitudes. PyTorch uses *inverted* dropout: it divides the surviving activations by (1 − p) during training, so the expected value is preserved and eval() mode can simply pass everything through unchanged with no rescaling at all.

The 1/(1−p) factor is the detail that makes the train/eval switch free, and knowing it prevents a whole class of confusion — notably 'why are my eval activations larger than my train activations' (they are not) and hand-written dropout that scales at the wrong time. It also explains why p=0.5 on a narrow layer is destructive: halving a 16-unit layer leaves 8 noisy units, while halving a 2048-unit layer is a mild perturbation.

Weight decay adds a penalty on the size of the weights, pulling every weight slightly toward zero at each step unless the loss gradient actively opposes it. This is the same L2 idea as Ridge regression from the ML syllabus, applied to a network's weight matrices. Two deep-learning-specific details matter. First, use AdamW rather than Adam: classical L2 inside Adam gets divided by the adaptive per-parameter scale, so the effective decay varies unpredictably per parameter, while AdamW applies it as a separate decoupled step. Second, exclude biases and normalization parameters — shrinking a BatchNorm γ toward zero attenuates the layer's output for no regularisation benefit.

It is the cheapest regulariser available: one hyperparameter, no extra compute, and no change to the forward pass. But it is routinely applied wrongly, and the two wrong applications — plain Adam and decaying every parameter — both quietly weaken it rather than failing loudly, so the tuning sweep that follows explores a distorted axis.

Augmentation regularises the *data* rather than the model: each epoch shows a randomly transformed version of every example, so the model never sees the identical input twice and cannot memorise pixel-exact training images. Crops, flips, colour jitter and random erasing for vision; mixup and cutmix, which blend two examples and their labels; token dropping or paraphrase for text; time warping and noise for audio. The transformations must preserve the label — and must be applied to the training set only, because a validated or tested model should be measured on real inputs.

It is the one regulariser that adds information rather than only removing capacity: it encodes what you know about the task's invariances. Telling the model 'a flipped cat is still a cat' is a genuine prior, which is why augmentation often beats dropout and weight decay on small image datasets. It is also the regulariser most often applied incorrectly, with transforms that silently destroy the label.

Early stopping tracks a validation metric each epoch, keeps the checkpoint that scored best, and halts once it has not improved for patience epochs. The ML syllabus introduced it as capacity control; a deep net adds two practical wrinkles. Validation curves are noisy, so patience=1 stops on ordinary fluctuation and wastes a run — a patience of 5-10 evaluations is normal. And because the final epoch is often not the best one, you must actually restore the saved best weights; a training loop that stops and then evaluates whatever is in memory reports the wrong model.

It does the work of a regulariser with no hyperparameter search over model capacity: the model is free to be large, and training simply stops at the epoch where it began to memorise rather than generalise. It is also the natural pairing for a warmup-and-decay schedule (module 8), where the last epochs make small refinements and the exact stopping point matters least.

Two curves and their gap say which of this module's tools to reach for. Training loss falling while validation loss flattens and then rises is memorisation, and the gap is its size — the case regularisation addresses. Both curves high and close together is underfitting, where adding dropout makes things strictly worse. Both falling together with a small stable gap is a healthy run that should simply continue. The gap is a ratio to judge in context, not a number with a universal threshold: a 0.05 gap is large when the loss is 0.1 and negligible when it is 3.

It decides direction. Regularisation removes effective capacity, so applying it to an underfitting model is actively harmful — and 'the model is not doing well, add dropout' is a common reflex. Making the diagnosis explicit before choosing the tool is the same discipline as module 10's triage, one layer up.