Deep Learning Fundamentals

A deep network is a parameterised function made by composing simple functions: input → layer → activation → layer → prediction. Each layer receives a representation, transforms it using learned weights, and passes a new representation onward. Training chooses the parameters that minimise a loss on examples; depth lets later layers build features from earlier ones, such as edges becoming shapes and then object parts. The same computational pattern can map pixels to labels, tokens to the next-token distribution, or noisy latents to an image.

This is the one picture behind image classifiers, language models, and diffusion models. Architecture changes which functions can be composed and which assumptions the model makes about the data; training changes the numerical parameters inside those functions. Keeping those two decisions separate makes debugging much clearer: a bad architecture limits what can be represented, while bad optimisation or data prevents a capable architecture from finding it.

Classical machine learning asks a person to decide what the model gets to see: edge histograms for images, n-gram counts for text, ratios and lags for tabular data. Deep learning keeps the raw signal and learns the intermediate representations as part of fitting the loss, so the features are chosen by optimisation rather than by intuition. The cost is that those learned features need data and compute to discover, and are not labelled in human terms afterwards.

It decides where your effort goes. On a small tabular dataset, an afternoon of feature engineering plus gradient boosting usually beats a neural network, because there is not enough signal to learn the features from scratch. On pixels, audio, or text, the hand-engineered route stalls: nobody can write down the features that separate ten thousand object classes, and the learned ones win by a wide margin.

Parameters are the tensors gradient descent writes to: every weight matrix and bias vector, plus embedding tables and normalisation scales. Hyperparameters are everything you fix before the run and the optimiser never touches: depth, width, activation choice, learning rate, batch size, weight decay, dropout probability, number of epochs. model.parameters() enumerates exactly the first group, which is why the optimiser is constructed from it.

The division tells you which knob to turn when a run disappoints. A model that cannot fit its own training batch usually has a hyperparameter problem — too little capacity, a learning rate that cannot move, a broken architecture. A model that fits training data and fails on validation data has enough parameters and the wrong hyperparameters around them. Confusing the two produces the classic wasted week: training longer to fix a choice that training cannot change.

Each layer can only apply one transformation, but a stack can apply a transformation to a transformation. In a vision model that shows up as a legible hierarchy — oriented edges, then corners and textures, then parts like wheels or eyes, then whole objects — where every level is built from combinations of the level below. The same compositional argument holds for language: characters into subwords, subwords into phrases, phrases into the referents a sentence is about.

Depth buys reuse. An 'eye' detector built from twenty edge detectors costs twenty weights plus a combination; a shallow model that must recognise an eye directly from pixels needs a separate template for every position, scale and lighting condition it will meet. That is why adding layers often beats adding width at equal parameter count, and why very wide shallow networks are both expensive and disappointing.

The universal approximation theorem says a network with one hidden layer, a non-polynomial activation and enough units can approximate any continuous function on a bounded region to any accuracy you like. It is a statement about existence: some setting of the weights is close enough. It says nothing about how many units 'enough' is, whether gradient descent can find that setting, or how many examples it would take to pin it down.

It is the most-cited and least-useful theorem in the field, and knowing why protects you from two bad arguments. The first is 'one hidden layer suffices, so depth is fashion' — true in principle, false in the width and sample size required. The second is 'the network can represent it, so training failure means bad data' — representability and learnability are separate questions, and almost every real failure is the second one.

A tensor is the NumPy ndarray mental model — a dense, regularly shaped block of numbers of one dtype — plus two things that matter in deep learning: a device saying which processor's memory holds it, and optionally an autograd history saying which operations produced it. Shape is the contract between layers, dtype decides precision and memory, device decides whether an operation is legal at all. Leading dimensions are conventionally batch-first in PyTorch: (batch, features), (batch, channels, height, width), (batch, tokens, model_dim).

Nearly every error in a deep learning session is one of these three fields being wrong, not the mathematics. A shape mismatch stops the run loudly; a dtype mismatch silently loses precision or refuses to index; a device mismatch raises the single most common runtime error in PyTorch. Reading a tensor as shape/dtype/device rather than as a list of numbers is what makes those errors immediately diagnosable.

Every training script, from a tutorial MLP to a frontier language model, is the same five objects: the data (a Dataset plus a DataLoader that batches and shuffles it), the model (an nn.Module holding the parameters), the loss (a function turning prediction and target into one scalar), the optimizer (the rule that turns gradients into parameter changes), and the loop that runs forward, loss, backward, step over and over. Anything else — schedulers, checkpoints, logging, mixed precision — hangs off one of those five.

It is the map for this whole topic, and the first question to ask of a failing run: which of the five is wrong? A loss that never moves is usually the loop (a missing backward, a missing step, a detached tensor). A loss that moves and validation that does not is usually the data. A loss that goes to NaN is usually the optimizer's learning rate or the loss's numerics. Naming the five keeps debugging a search over five suspects instead of a vague feeling.

Deep learning wins where the input is high-dimensional, raw and structured — pixels, waveforms, tokens, molecules — and where there is enough labelled or self-supervised data to discover the structure. It reliably loses on small, heterogeneous, well-understood tabular data, where gradient boosting on engineered features remains the strong baseline. Between those poles the honest answer comes from an experiment, not from a preference.

Choosing the family is a bigger decision than any hyperparameter inside it, and it is made before you can measure anything. Getting it wrong costs weeks: a neural network on 800 rows of tabular data will underperform a tree ensemble you could have fitted in a minute, and a gradient-boosted model on raw pixels will underperform a small CNN by a margin no tuning closes.