MLPs

An MLP alternates fully connected layers and nonlinearities. Every output unit in a dense layer can use every input feature, so the model can learn rich interactions among a fixed set of values. It has no built-in notion of image locality, token order, or graph structure; those properties must be encoded in the input or supplied by a different architecture. Width controls how many features a layer can represent, while depth lets the network compose those features into more complex functions.

MLPs are the right baseline for fixed-length vectors and the feed-forward sublayer inside Transformers. They are usually the wrong inductive bias for raw images and long sequences because they do not share weights across spatial positions or time. A simple MLP baseline is still valuable: if a specialised model cannot beat it honestly, the extra complexity has not earned its cost.

There is no MLP-specific mathematics. Take nn.Linear from module 2, a nonlinearity from module 3 — without which the whole stack collapses algebraically to one linear map — a loss from module 5, an optimiser from module 8, and optionally the normalization and dropout of module 11. Stack at least one hidden layer and you have a multilayer perceptron. The only architectural decisions are the number of hidden layers, their widths, which activation to use, and where to place normalization and dropout. Everything else was already decided by the task: the input width by the feature count, the output width and the loss by what you are predicting.

Naming an MLP as an assembly rather than a new object makes the whole first half of this topic pay off at once, and makes the later architectures legible: a CNN replaces Linear with a weight-sharing operator, an RNN reuses one layer across timesteps, and a Transformer block wraps attention around an MLP. The pattern never changes — only which operator mixes information across positions.

A complete supervised run on a fixed-length input has six parts and no more: a Dataset/DataLoader that yields batches, a model, a loss, an optimiser, a training loop that calls zero_grad/backward/step, and a validation pass in eval() mode. Two shape contracts must hold for classification: the logits must be (batch, n_classes) and the targets (batch,) of dtype long holding class indices. Standardise the features using statistics from the training split only — an MLP is scale-sensitive in a way tree models are not, and a feature measured in millions will dominate the first layer's gradients.

This loop is the reference implementation the rest of the topic varies. Getting it written once, correctly, means every later architecture is a change to one line — the model — rather than a new training script with a new set of bugs. It is also the shortest path to a real baseline number, which every subsequent architectural claim has to beat.

The universal approximation theorem says a single hidden layer with enough units can approximate any continuous function on a bounded region to any accuracy. Read carefully, it is an existence statement about weights, and it is silent on all three things that decide whether you can train the model: how many units 'enough' is (it can be exponential in the input dimension), whether gradient descent from a random start will find those weights, and how many labelled examples you would need to identify them from data rather than from the function itself.

It is the most-cited and least-useful theorem in the field, and misreading it produces a specific bad habit: widening one hidden layer indefinitely and expecting the fit to improve. Depth, weight sharing, and architectural priors matter precisely because representability was never the binding constraint — optimisation and sample efficiency are. A CNN does not beat an MLP on images because it can represent more functions; it beats it because it can *learn* the right one from the data available.

Width sets how many features one layer can hold in parallel; depth sets how many times those features can be recombined into higher-level ones. They cost differently. A hidden layer of width w between two others contributes on the order of w² parameters, so doubling width roughly quadruples that layer's parameters and its matrix-multiply cost. Adding a layer at the same width grows parameters linearly and depth-related optimisation difficulty non-linearly — which is where module 10's vanishing gradients, residual connections, and normalization become the deciding factors rather than an afterthought.

It turns 'make the model bigger' into two separate, differently-priced decisions. For tabular data, 2-4 hidden layers of a few hundred units is usually the whole useful search space, and time is better spent on features and regularisation. Knowing the w² scaling also explains why the first layer after flattening an image dominates a naive model's parameter count.

A dense layer takes a flat vector, so an image must be flattened first — and flattening throws away the fact that pixel (10, 10) is adjacent to (10, 11). Two consequences follow. The parameter count explodes: a single Linear from a 224×224×3 image to 1000 hidden units needs 224·224·3·1000 ≈ 150 million weights, more than a whole ResNet-50. And there is no weight sharing, so a pattern learned at the top-left is stored in an entirely separate set of weights from the same pattern at the bottom-right — the model must learn every feature once per location, and has no way to generalise a shifted example.

This is the precise motivation for module 13. A convolution keeps the dense-layer idea and adds two assumptions — locality (a unit looks at a small neighbourhood) and translation-equivariant weight sharing (the same kernel is applied everywhere) — which together cut that 150 million to a few thousand while making a shifted cat recognisable. The architecture is not more powerful; it is better matched to the structure of the data.

Two module-3 and module-10 failures show up concretely here. A ReLU unit whose pre-activation is negative for every training example outputs zero always, receives exactly zero gradient always, and is permanently dead — a common outcome of a learning rate large enough to push a bias far negative in one step. Separately, a too-small initialization makes each layer shrink the activations, so by layer 8 the representations are numerically negligible and the output is effectively constant. Both look identical from the loss curve: a plateau. They are trivially distinguishable by measuring activations.

An MLP is the simplest place to build the habit of measuring activations rather than inferring them. Two numbers per layer — the fraction of units that are zero for *every* example in a batch, and the activation standard deviation — separate a dead-unit problem (lower the learning rate, try LeakyReLU or GELU, check the bias initialization) from a scale problem (fix the initializer, add normalization) in one forward pass.

Each nn.Linear(a, b) holds a (b, a) weight matrix and a (b,) bias, so a·b + b parameters, and maps (batch, a) to (batch, b). Walking that stage by stage answers most practical questions before training: where the parameters actually are (almost always the widest transition, which is why the first layer after a flatten dominates), whether the shapes chain correctly, and roughly how much memory activations will need. print(model) gives the structure and a two-line loop gives the counts — both are cheaper than discovering a shape mismatch at the last layer of a long forward pass.

Parameter counts are how architectural claims get checked. 'A deeper model' and 'a wider model' are meaningless comparisons unless the counts are stated, and 'this CNN beats the MLP' is only interesting if the CNN is not simply ten times larger. Making the table a habit also makes module 13's parameter-efficiency argument and module 18's scaling table read as arithmetic rather than assertion.