Initialization & Gradient Problems

Every layer scales both forward activations and backward gradients. When many scales below one multiply together, early layers receive vanishing gradients; when scales above one compound, values or updates explode. Xavier initialization chooses variance based on fan-in and fan-out for roughly variance-preserving tanh-like layers, while He initialization is calibrated for ReLU-like layers that discard half their signal. Gradient clipping is a safety mechanism for occasional extreme updates, not a substitute for a consistently ill-scaled model.

Bad initialization can make a correct model appear broken before it has learned anything. These are signal-scale problems, not evidence that the dataset is impossible. Watching activation and gradient distributions by layer can distinguish a structural scaling failure from ordinary underfitting, and is especially valuable in deep recurrent or Transformer training.

Weights must start at *different* random values, not merely small ones. If every weight in a layer holds the same number, every neuron in that layer computes the same pre-activation, receives the same gradient from backpropagation, and is updated by the same amount — so it stays identical to its neighbours forever. A 512-unit layer initialized to a constant has the representational capacity of exactly one unit, no matter how long it trains. Random initialization is what makes the units distinguishable in the first place; biases can safely start at zero because the weights already break the tie.

This is the one initialization failure that produces no error, no NaN, and no warning — just a model whose loss falls to the level of a much smaller model and then stops. Knowing the mechanism turns a mystifying plateau into a one-line check: if the rows of a weight matrix are identical (or become identical), the layer is collapsed.

Backpropagation (module 7) builds each early-layer gradient as a product of one factor per layer between it and the loss. A network 30 layers deep multiplies roughly 30 such factors together. If each factor averages 0.8, the product is 0.8³⁰ ≈ 0.001; at 0.5 it is under 10⁻⁹. Nothing is broken — the chain rule is simply doing arithmetic — but the first layers receive a gradient so small that any reasonable learning rate leaves them effectively frozen at their random initialization while the last few layers train normally.

It explains the signature symptom of an untrainable deep network: the loss falls a little, then flattens well above where a shallower model lands. The fix is never 'train longer' — a gradient of 10⁻⁹ does not accumulate into progress. It is to change the factors: a better initialization scale, a non-saturating activation, normalization, or a residual path that skips the product entirely.

The same product runs the other way when the per-layer factors exceed one: 1.5³⁰ ≈ 190,000. The symptoms are louder than vanishing — a loss that spikes from 2.1 to 40,000 in one step, then becomes nan and stays there, because one enormous update threw every parameter far outside the region where the forward pass is finite. Recurrent networks are the classic victims, since the same weight matrix is applied once per timestep, but any deep stack with a slightly-too-large initialization or learning rate can do it.

A nan loss is unrecoverable: once parameters are nan, every subsequent gradient is nan, and the run is dead however many epochs remain. Recognising the spike *before* the nan — and knowing it is a scale problem rather than a data problem — is the difference between a two-minute fix and a day spent auditing the dataset.

Xavier initialization draws weights with variance 2 / (fan_in + fan_out), which is the scale that keeps a layer's output variance roughly equal to its input variance for activations that are near-linear around zero — tanh and sigmoid. The derivation is short: a unit summing fan_in independent inputs multiplies the input variance by fan_in × Var(w), so setting Var(w) ≈ 1/fan_in makes that factor one. Averaging fan-in and fan-out balances the forward pass against the backward pass, which sees the transposed matrix and therefore the opposite fan.

It is the concrete answer to 'how random is random enough'. Too small a variance shrinks the activations layer by layer until the signal is numerically negligible; too large saturates tanh, where the derivative is near zero and the gradient dies for a different reason. Xavier picks the one scale where a 20-layer tanh network still has recognisable activations at layer 20.

ReLU sets every negative pre-activation to exactly zero, which removes about half the signal's variance. Xavier's scale, derived for a near-linear activation, therefore undershoots by a factor of two per layer for a ReLU stack — and a factor of two per layer compounds exactly like every other factor in this module. He initialization corrects it with variance 2 / fan_in, the extra 2 paying for the half that ReLU discards. nn.init.kaiming_normal_(w, nonlinearity="relu") is the direct form; calculate_gain supplies the equivalent multiplier for leaky ReLU and other variants.

This is why the default choice of initializer depends on the activation, not on the architecture. In a 30-layer ReLU network, Xavier's missing factor of two becomes 2⁻³⁰ — the difference between activations of order one at the output and activations of order 10⁻⁹. Matching the initializer to the nonlinearity is a one-line change that decides whether the network trains.

Module 6 covers the mechanics — clip_grad_norm_ rescales the gradient by its global norm between backward() and step(). clip_grad_value_ is the blunter alternative: it clamps each element independently, which does change the direction rather than only its length.

Exploding gradients are not evenly distributed across architectures: a recurrent model applies the same weight matrix at every timestep (module 15), so one unstable eigenvalue compounds over the whole sequence length rather than over a fixed handful of layers. Clipping is close to mandatory for RNNs specifically, in a way it usually isn't for a feedforward network with reasonable initialization.

A residual block computes y = x + F(x) rather than y = F(x). On the backward pass the addition sends the incoming gradient down two routes: through F, where it picks up the block's factor, and straight through the +, where it passes unchanged with a local derivative of exactly 1. A 50-block residual network therefore always has one path from the loss to layer 0 whose gradient is not multiplied by anything, no matter what the 50 blocks do to their own copy. This is why depth stopped being a barrier: skip connections remove the product, rather than tuning its factors.

It reframes depth. Without residuals, each added block multiplies another factor into every earlier gradient and makes training harder; with them, a block can learn F(x) ≈ 0 and become a near-identity, so adding capacity is close to free. Every modern deep architecture — ResNets, Transformer blocks (module 18), diffusion U-Nets — is built on this one property.

The symptoms of this module's failures overlap, so the diagnosis is a short ordered checklist rather than a guess. Loss immediately nan: exploding — lower the learning rate, add clipping, check for a division or log(0) in the loss. Loss flat from step 1 at exactly ln(n_classes): nothing is reaching the early layers — check per-layer gradient norms, the initializer against the activation, and whether the blocks are residual. Loss falls then stalls far above a shallower baseline: depth is hurting, which is the residual-connection case. Some layers' weights never change: dead ReLUs (module 3) or a collapsed symmetric layer. Each branch names a different fix, and applying the wrong one wastes a full training run.

The instinct when a deep model will not train is to change everything at once — optimiser, learning rate, architecture, batch size — which guarantees that whatever happens next teaches you nothing (module 25). Three cheap measurements — per-layer gradient norm, per-layer activation std, and the fraction of parameters that actually moved after one step — distinguish all four cases before you change any hyperparameter.