Transformers

A Transformer stacks blocks containing multi-head self-attention and a position-wise MLP, each wrapped in residual connections and normalization. Attention lets tokens exchange information; the MLP then transforms each token independently using its newly contextualised representation. Residual additions preserve the incoming representation beside the block's new contribution, while normalization keeps layer scales manageable. Positional embeddings or encodings supply order because attention alone is permutation-invariant; an unpositioned model sees the same set of token vectors regardless of order.

Residual paths preserve an easy route for signals and gradients through many layers; position is what lets the model distinguish 'dog bites man' from the reverse. This repeated block structure also makes Transformer configuration legible: depth sets how often contextualisation is refined, width sets representation capacity, head count divides attention into multiple subspaces, and context length sets the available evidence and computational bill.

Written out, one pre-norm block is four lines and every line came from an earlier module. h = x + Attn(LN(x)) mixes information *across* positions — module 17's self-attention, wrapped in module 10's residual and module 11's LayerNorm. y = h + MLP(LN(h)) then transforms *each* position — module 12's MLP, wrapped identically. The two sublayers alternate roles deliberately: attention is the only operation in the whole architecture that moves information between positions, and the MLP is the only one that gives each position substantial nonlinear processing of its own. Everything a Transformer computes is those two steps repeated.

Seeing the block as an assembly rather than a monolith is what makes it modifiable and debuggable. Almost every variant in the literature is a substitution into one of these slots — a different attention (sparse, sliding-window, linear), a different feedforward (gated, mixture-of-experts), a different norm (RMSNorm), a different position scheme — and the surrounding structure is unchanged. It also makes the shapes checkable: the block is shape-preserving, so (batch, n, d_model) goes in and comes out.

The MLP inside a block is a plain two-layer network — expand d_model to typically 4 × d_model, apply a nonlinearity, project back — applied identically and independently to every position, with the same weights at every position. It exists because attention, on its own, only takes weighted averages: it can gather the right information from elsewhere in the sequence but performs little nonlinear computation on it. The MLP is where that gathered representation is actually transformed. Its 4× expansion means it holds roughly two thirds of a transformer's parameters, which is why mixture-of-experts architectures target this sublayer specifically when scaling parameter count without scaling compute per token.

It answers a question the attention-centred presentation leaves open — what the rest of a Transformer is for. The division of labour is clean and worth keeping: attention routes, the MLP computes. It also explains a shape fact that surprises people, which is that the widest tensors in a Transformer are inside the feedforward sublayer rather than in attention.

Three families, differing only in which sequences exist and which masks apply. An encoder-only model (BERT-style) runs bidirectional self-attention over one sequence with no causal mask, so every token sees the whole input; it is trained with a masked-token objective and suits classification, retrieval, tagging and embedding. A decoder-only model (GPT-style) runs causal self-attention over one sequence, predicts the next token at every position, and generates autoregressively. An encoder-decoder (T5-style, and the original 2017 architecture) encodes a source bidirectionally and decodes causally, with cross-attention from the decoder into the encoded source — the natural shape for translation and summarisation, where input and output are distinct sequences of different lengths.

The choice follows from the task's information structure, not from fashion: if the whole input is available and you need a representation, an encoder is right and a causal mask would only hide legitimate context; if you must produce tokens one at a time, a decoder is required. Knowing the families also makes checkpoints legible — a model's name usually tells you which one it is, and therefore what it can and cannot be asked to do.

Three properties compound. Every sublayer is parallel over the sequence (module 17), so a training step's wall-clock time is set by available compute rather than by sequence length — unlike an RNN's n dependent steps (module 15). The residual stream (module 10) makes very deep stacks trainable, so depth is a usable axis rather than a barrier. And the architecture is uniform: the same block repeated, with no per-layer design work, so scaling is a change to three integers — layers, heads, width. Together these make the relationship between compute, data and loss smooth and predictable enough to extrapolate, which is what scaling-law work depends on.

It explains the historical outcome rather than asserting it. Recurrence was not displaced because attention models language better in principle, but because a fixed hardware budget buys far more training on a parallel architecture, and the models that could consume enormous corpora within a reasonable wall-clock budget were the ones that got large enough to be surprising. It also sets the practical limit: the same argument does not apply to *inference*, which is still one token at a time.

A block's parameters are dominated by two terms: attention's four d_model × d_model projections, 4d², and the feedforward's two d_model × 4d_model matrices, 8d². So a block is roughly 12 · d_model², and a model is n_layers × 12 · d_model² plus the embedding table, vocab × d_model. Two consequences are worth internalising. Width is quadratic and depth is linear, so doubling d_model costs four times as much as it appears to while doubling the layer count costs exactly double. And head count does not enter the formula at all: 8 heads of 64 and 16 heads of 32 have identical parameter counts, because the head dimension is d_model / n_heads.

It is how a configuration gets sanity-checked before a run. It answers 'will this fit in memory', it makes two models comparable ('deeper' and 'wider' are only meaningful next to a count), and it locates the surprises — that heads are free, that the embedding table dominates a small model with a large vocabulary, and that width is the expensive axis.

The concrete failure that ties this group back to module 10. Take a correct block, delete the two x + additions so it reads x = MLP(LN(Attn(LN(x)))), and stack it 24 times. Nothing errors and the shapes all agree. What happens is that the gradient to the early blocks now passes through 48 transformations with nothing to bypass them, module 10's product collapses, and the loss falls briefly — the last few blocks are still learning — and then flattens far above where the residual version lands. With post-norm and no warmup the same deletion can instead diverge outright. In either case the model is untrainable at depth, and the cause is two missing characters.

It is the most valuable thing to know about transformer depth: the blocks are not what makes a deep stack work, the residual stream is. Reading the architecture as 'a residual stream that blocks read from and write into' also explains a great deal of interpretability work, and makes it obvious why the block must be shape-preserving — a residual add requires its two operands to have the same shape.

Six numbers describe one: n_layers, d_model, n_heads, the feedforward multiplier (nearly always 4), vocab_size, and max_seq_len. Four constraints tie them together. d_model must be divisible by n_heads. The head dimension d_model / n_heads is usually kept at 64-128, so heads and width move together. max_seq_len sets the quadratic attention cost and, for learned positional embeddings, a hard ceiling the model cannot exceed. And the block must be shape-preserving, which the residual additions require. Checking those four before a long run is minutes of work against hours of wasted compute.

Almost every 'why does this not work' in a hand-written Transformer is one of these four, and three of them fail loudly while the fourth — exceeding max_seq_len — can silently index a positional table out of its trained range. Being able to state a model as six numbers is also what makes two models comparable and a configuration reviewable.