Gradient Descent

A gradient gives the local direction in which each parameter increases loss, including both direction and sensitivity. Gradient descent moves the opposite way: θ ← θ − learning_rate × ∇θ loss. In practice, a mini-batch estimates that direction from a small random sample of examples; the estimate is noisy, but is much cheaper than computing the full dataset gradient and that noise can help escape unhelpful local geometry. One optimiser step updates every trainable parameter once after backpropagation has populated its gradient.

The learning rate controls the basic trade-off: too high diverges or bounces past a solution; too low makes progress impractically slow. This is why a loss curve is a diagnostic instrument, not merely a score: smooth stagnation, erratic spikes, and steady decline call for different interventions. Mini-batch size also changes the noise and number of updates per epoch, so it cannot be tuned independently of the learning rate.

Batch (or full-batch) gradient descent computes the loss over the entire training set before taking one step, so each step is an exact gradient and epochs contain exactly one update. Stochastic gradient descent uses one example per step: extremely noisy, but thousands of updates per epoch. Mini-batch descent — what everyone actually means by 'SGD' in PyTorch — uses 32 to 4096 examples per step, which is enough to average out most of the noise while still fitting in memory and giving many updates per epoch.

Batch size trades three things at once: gradient quality, updates per epoch, and hardware utilisation. Tiny batches give noisy directions but many cheap steps, and the noise itself acts as a regulariser. Huge batches give near-exact directions and excellent GPU throughput, but far fewer steps per epoch, and the extra precision buys less than the lost update count — which is why very large-batch training needs a scaled-up learning rate and a warmup (module 8) to stay competitive.

The learning rate multiplies every gradient before it is subtracted, so it sets the length of every step the model takes. Too small and the loss falls but never arrives within your compute budget; too large and each step overshoots the valley it was aiming at, so the loss oscillates, plateaus high, or diverges to NaN. The usable range for a given model and loss usually spans about an order of magnitude, and it moves when you change the architecture, the batch size, or the normalisation.

Nothing else in deep learning has this much leverage per minute spent. A learning rate an order of magnitude too low turns a 20-minute run into a week; one an order of magnitude too high makes a perfectly good architecture look broken. This is also why the shape of the loss curve is a diagnosis rather than a score: a flat high curve, a noisy sideways curve, and a curve that spikes to NaN each point at a different correction.

A critical point is where the gradient is zero. In one dimension that is a minimum or a maximum, but in a space with millions of parameters a zero-gradient point is overwhelmingly likely to be a saddle: curving up in some directions and down in others. True local minima that are much worse than the global one are rare in large networks, because a point being a local minimum requires the curvature to be positive in *every* one of a million directions at once.

It changes what you worry about. The folk story that deep learning is a struggle against bad local minima is largely wrong; the real obstacles are saddle points and long flat plateaus, where the gradient is tiny and progress stalls without anything being stuck. That diagnosis points at different fixes — momentum to carry through flat regions, adaptive per-parameter step sizes to move along directions the gradient barely mentions, and a higher learning rate or a warm restart to escape a plateau — rather than at random restarts.

Instead of stepping along the current batch's gradient, momentum keeps a running average of recent gradients — a velocity — and steps along that. Directions the gradients agree on accumulate and the step grows; directions where consecutive batches disagree cancel out and the step shrinks. In PyTorch it is one argument: torch.optim.SGD(params, lr=0.1, momentum=0.9).

It is the cheapest fix for the two problems above. Across a flat plateau, plain descent takes tiny steps forever while momentum carries speed from before the plateau; in a narrow valley, plain descent zigzags across the walls while momentum cancels the crosswise oscillation and accelerates along the floor. The effect on wall-clock time is often larger than a careful learning rate search, which is why momentum=0.9 is a sensible default rather than a tuning opportunity. Module 8 unpacks how it works and what Adam adds on top.

Clipping rescales the gradient before the optimiser sees it, so a rare enormous gradient cannot produce an enormous step. clip_grad_norm_ computes the global norm across all parameters and, if it exceeds a threshold, scales every gradient down by the same factor — preserving the direction and capping only the length. It goes between loss.backward() and optimizer.step(), and a threshold of 1.0 is a common default in language-model training.

Loss surfaces have cliffs. One unlucky batch — an outlier, a long sequence, a badly scaled input — can produce a gradient several orders of magnitude larger than typical, and one step along it destroys hours of training in a single update, usually surfacing as a sudden loss spike or NaN. Clipping makes that failure mode impossible without changing anything about normal steps. It is a safety net, though, not a cure: if clipping triggers on most batches, the learning rate or the input scaling is wrong.

Mark a tensor with requires_grad=True, compute a scalar from it, and call .backward(): PyTorch fills each participating tensor's .grad field with the derivative of that scalar with respect to it. For a model, model.parameters() are already marked, so loss.backward() populates param.grad for every one of them. Those gradient tensors have exactly the same shape as the parameters — one derivative per weight.

Being able to print a gradient turns most training mysteries into a two-line check. A .grad that is None means the parameter never participated in the loss — a detached tensor, an unregistered layer, a frozen requires_grad=False. All-zero gradients mean a dead activation or a loss with no dependence on that parameter. NaN gradients localise the numerical problem to a specific layer. The per-parameter gradient norm is also the most useful training diagnostic after the loss itself (module 9).

An optimiser is not magic; plain SGD is two lines. After backward() has filled every .grad, walk the parameters and subtract lr * p.grad from each, inside torch.no_grad() so the update itself is not recorded as part of the graph. Then zero the gradients, because PyTorch accumulates them (module 7). torch.optim.SGD does exactly this, plus momentum and weight decay.

Writing it once removes the last piece of mystery from the training loop, and it makes the two most common loop bugs obvious. Updating without no_grad() builds a graph over the update, which leaks memory and eventually errors; forgetting to zero the gradients means every step uses the sum of all gradients so far, which grows the effective step size epoch after epoch. Both are visible in these five lines in a way they are not inside optimizer.step().