Debugging Neural Networks

First verify that a tiny, fixed batch can be overfit: loss should approach zero and predictions should become correct. This deliberately removes generalisation, throughput, and data-diversity questions so a failure points to labels, shapes, objective, parameter registration, gradients, or update code. Once it passes, inspect inputs and labels, loss scale, gradient norms, parameter deltas, activation distributions, and train-versus-validation curves one hypothesis at a time. Log examples and intermediate outputs as well as scalar metrics; a plausible average can hide systematically broken preprocessing.

A large training run confounds every failure mode. A tiny controlled test separates broken code or labels from an honest capacity or generalisation problem. It also creates a regression test for the training system: when a refactor breaks the ability to overfit one batch, stop before spending hours or GPU budget on a full run.

Before any training, a correctly built classifier should be maximally uncertain, which gives a known cross-entropy: ln(n_classes). Ten classes should start near 2.303, two classes near 0.693, a 50,000-token vocabulary near 10.82. For regression with standardised targets, the initial MSE should be near the target variance. These are arithmetic predictions, not rules of thumb, and they can be checked in one forward pass.

It is the cheapest bug detector in deep learning and almost nobody runs it. An initial loss far *above* the prediction means the outputs are not near-uniform — usually a bad initialisation scale, or a double-applied activation such as a softmax before CrossEntropyLoss. An initial loss far *below* it is more alarming: the model already knows something before training, which means label information has reached the input, a target has been concatenated into the features, or the loss is being computed against the wrong tensor. Both diagnoses are available in seconds, before a single GPU-hour is spent.

A loss curve has a small number of recognisable shapes and each points at a different cause. Flat from the start means no learning signal reaches the parameters: the learning rate is far too low, part of the graph is detached or frozen, or the optimiser was constructed over the wrong parameter list. NaN or inf means numerical instability: the learning rate is too high, or a log/divide/sqrt met zero. Violent oscillation without progress means too large a step or too small a batch. A smooth fall then a plateau is normal and is a scheduling question.

Treating the curve as a score rather than an instrument is what makes training feel like guesswork. Each of those shapes has a different first intervention, and they are not interchangeable: raising the learning rate fixes flat and causes NaN; lowering it fixes NaN and causes flat. Reading the shape first turns a broad search over hyperparameters into a specific hypothesis, which is the difference between an afternoon and a week. The shape also tells you *when* to look — a curve that was healthy and then diverged at step 4000 is a data or numerics event at step 4000, not a configuration problem.

The loss is a summary and it can look reasonable while the gradients are wrong. Inspecting them directly answers three questions the loss cannot: does every parameter that should be learning actually have a .grad, are the magnitudes in a workable range, and are any of them NaN. A parameter whose gradient is None after backward() is not connected to the loss; one whose gradient norm is 1e-9 is effectively frozen; one at 1e4 will be clipped or will blow up.

It localises the failure to a layer, which the loss never can. A per-layer gradient-norm log is the fastest way to find a vanishing-gradient region, a detached branch, an unused head, or a normalisation layer that has collapsed. It is also the only way to confirm the optimiser is doing anything: comparing parameter values before and after step() proves the update path works end to end, which is a different claim from 'the loss went down' — a loss can fall because of a batch-norm running statistic while the weights sit still.

Shape errors are the most common class of deep-learning bug and the error message usually names the wrong place. PyTorch reports the operation that failed, which is where two incompatible shapes finally met — often several layers after the one that produced the wrong shape. The systematic fix is to make every layer boundary print its input and output shape, then read down the list for the first row that is not what you intended.

Reading shapes at every boundary converts a search into a scan. It also catches the far more dangerous case: a shape that is wrong but *compatible*, so nothing raises. A (32, 1) target broadcasting against a (32,) prediction silently produces a (32, 32) loss matrix and trains on nonsense; a transposed (batch, seq) versus (seq, batch) tensor runs perfectly and learns across the wrong axis. Those bugs have no error message at all, and a boundary-by-boundary shape log is how they are found.

Three categories, three completely different investigations. A data bug means the tensors are wrong — misaligned labels, a broken normalisation, a corrupted decode, an augmentation applied to the image but not to the mask. A model bug means the architecture cannot express or propagate what is needed — a missing nonlinearity, a wrong mask, a bottleneck too narrow, a normalisation in the wrong place. A training-loop bug means the machinery around a correct model is wrong — a missing zero_grad, a mode left in eval, a scheduler stepping at the wrong cadence, a loss computed against the wrong tensor.

Separating them is what stops debugging turning into random perturbation. Each category has a distinct cheapest test. For data: look at the actual decoded examples and their labels, as images or text, not as statistics — a plausible mean hides an entirely broken preprocessing chain. For the model: does the one-batch test pass? If yes, the architecture can represent the answer and the model is not the problem. For the loop: do the parameters change, once, in the direction the gradient says? Running these three tests in order — data, model, loop — costs minutes and eliminates two thirds of the search space.

A validation score far above what the task plausibly allows is evidence of leakage, not of success. Deep learning adds leak paths that tabular ML does not have. Splitting *after* augmentation puts rotated copies of the same training image in both sets. Computing normalisation statistics over the whole dataset lets test-set means into training. Frames from one video, crops from one scan, or utterances from one speaker landing on both sides of the split makes validation a memorisation test. Deduplicating text after splitting leaves near-identical documents in both.

The ML syllabus established leakage as a category; the deep-learning versions are worth naming separately because the mechanisms are different and the augmentation one in particular is unique to this setting. It is also more dangerous here, because the symptom is a *good* number and nothing in the training loop complains. The defence is structural: split by the correct *group* — patient, video, document, speaker — before any transformation touches the data, and fit every statistic on the training split alone. A leaked split cannot be detected from inside the run; it can only be prevented by where the split happens.

The NumPy syllabus covered seeding a generator. Deep learning on a GPU has four more sources of variation. DataLoader workers each need their seed set. cuDNN benchmarks several convolution algorithms and picks the fastest, which may differ between runs. Many CUDA kernels — scatter-adds, some backward passes, atomic reductions — accumulate in nondeterministic float order, so the same inputs give bitwise different outputs. And multi-GPU gradient all-reduce adds its own ordering.

It matters because 'I changed one thing and the result moved' is only informative if the baseline is stable. Without determinism you cannot tell a real 0.4% improvement from run-to-run noise, so every comparison needs multiple seeds and a variance estimate — which is more expensive than making the run deterministic in the first place. The honest position is a choice, made explicitly: full determinism for debugging and for comparisons that must be exact, accepting a real slowdown; benchmark mode for production training, accepting that results are reproducible only in distribution and reporting a mean over seeds rather than a single number.