Attention

Attention is a learned retrieval operation. A query vector is compared with key vectors to produce relevance scores; softmax turns those scores into non-negative weights that sum to one; the output is the weighted combination of value vectors. A query asks what is relevant now, keys say what each candidate offers, and values contain the information to retrieve. The query, key, and value projections are learned, so the model can discover several useful notions of relevance rather than relying on a fixed distance rule.

Unlike a fixed recurrent state, attention can directly connect distant items, and its weights change with the current query. This creates short information paths between related positions, but it also means the cost and behaviour of the model depend on sequence length, masking, and the learned representations rather than on word distance alone.

Before any matrix appears, the three roles are worth stating as three questions. A query is 'what am I looking for right now' — produced from the position doing the looking. A key is 'what do I have on offer' — produced from each candidate position, and compared against the query to decide relevance. A value is 'what gets handed over if I am chosen' — also produced from each candidate, but serving a different purpose, which is why keys and values are separate projections of the same vector rather than the same tensor. The analogy is a lookup that returns a blend: instead of matching one key exactly, every key gets a score, and the answer is the score-weighted mixture of all the values.

Keeping the three roles distinct is what makes every later variant readable. Cross-attention is 'queries from the decoder, keys and values from the encoder'; self-attention is 'all three from the same sequence'; a key/value cache is 'the keys and values are reusable across steps but the query is new each step'. Each is one sentence once the roles are clear, and impenetrable otherwise.

A vanilla sequence-to-sequence RNN encodes the source sentence into its final hidden state and hands that one fixed-size vector to the decoder, which must generate the entire output from it. For a five-word sentence this is workable; for a forty-word one, a single 512-dimensional vector has to carry every word, its order and its structure, and translation quality falls sharply with source length. Attention removed the constraint: keep *all* the encoder states, and let each decoder step compute a query and take a weighted average over them. The decoder can then look at the part of the source it currently needs, and the alignment between source and target positions emerges from training rather than being engineered.

This is the origin story, and it explains the shape of the mechanism — attention is a fix for a compression bottleneck, which is why its output is an average over retained states rather than a transformation of a summary. It also explains why attention was a decoder-to-encoder operation for years before anyone applied it within a single sequence; module 17's self-attention is the later, and more radical, step.

Attention(Q, K, V) = softmax(QKᵀ / √d_k) V, read right to left in four steps. QKᵀ is every query against every key, giving an (n_q × n_k) matrix of raw compatibility scores — a dot product, so it is large when two vectors point the same way. Dividing by √d_k rescales those scores. softmax over the last axis turns each row into a probability distribution across the candidate positions. Multiplying by V replaces each row with the corresponding weighted average of the value vectors. The scaling exists because a dot product of two d_k-dimensional random vectors has variance proportional to d_k, so at d_k = 64 the raw scores are roughly 8× larger than at d_k = 1; feeding those into softmax saturates it into a near one-hot distribution whose gradient is almost zero, and the model cannot learn to adjust its attention. Dividing by √d_k puts the scores back in the range where softmax has usable gradients.

The √d_k is the one term in the formula that looks arbitrary and is not: it is a variance correction, and without it a wide-head model does not train. Knowing why also generalises — it is the same saturation argument as module 3's warning about sigmoid, and it is why attention logits are sometimes clamped or normalised further in very large models.

The softmax is taken over the *candidate* axis, so every query's weights are non-negative and sum to exactly 1 — the same construction as module 3's softmax over classes, with positions in place of classes. Two properties follow. The output is a convex combination of the value vectors, so it lies inside their span and cannot be larger in magnitude than the largest of them, which is what makes attention numerically stable. And the distribution's *shape* is meaningful: a peaked row means the query found one clearly relevant position, while a near-uniform row means it found nothing distinguishing and is returning an average. Its entropy is a cheap, real diagnostic.

It is the fact to check first when attention is not behaving. Rows that sum to something other than 1 mean the softmax is on the wrong axis — a genuinely common bug, since dim=-1 and dim=-2 both run without error and produce plausible-looking matrices. Uniformly flat rows across all heads and layers mean the scores carry no signal, which points at the scaling, the initialization, or a mask that removed everything.

One attention operation produces one distribution per query, so it can express one notion of relevance at a time. Multi-head attention splits the model dimension into h smaller subspaces — 8 heads of 64 dimensions rather than one of 512 — runs the whole scaled dot-product operation independently in each, concatenates the results, and applies one output projection. The cost is essentially unchanged because the head dimension shrinks proportionally, and the gain is that different heads can settle on different relationships: one tracking syntactic dependencies, another matching a pronoun to its antecedent, another simply attending to the previous token.

Without heads, a single softmax has to serve every kind of relationship at once, and the averaging blurs them. Heads are also the practical knob in a Transformer's configuration alongside depth and width (module 18), and the reason d_model must be divisible by n_heads — a constraint that otherwise looks like an arbitrary API requirement.

One row of an attention matrix is one query's distribution over the candidates, and it is the most interpretable object in a Transformer — with real limits. A row is read as: which positions carry weight, how concentrated it is, and whether the pattern is consistent across examples or an artefact of one. Many heads turn out to do something mundane and useful, such as attending almost entirely to the previous token or to a sentence-initial position that acts as a no-op sink. Both are genuine findings; neither is an explanation of the model's output, because the row describes one routing step among dozens of layers and heads.

Attention maps are the standard illustration of what a language model is 'looking at', and they are routinely over-read. Knowing what a row does and does not license — it shows which value vectors were mixed at this step, not which input caused the final prediction — is the difference between a useful diagnostic and a misleading visualisation.

QKᵀ has one entry per query-key pair, so for a sequence of length n it is an n × n matrix that must be computed, softmaxed, and multiplied by V — O(n² · d) work, and in a naive implementation O(n²) memory per head per layer. The consequence is stark: doubling the context length quadruples the attention cost. At n = 512 the matrix is 262k entries; at n = 32,768 it is a billion, per head per layer. This is why context length is the expensive dimension of a language model, why FlashAttention (which never materialises the full matrix) matters so much in practice, and why sparse, linear and sliding-window attention variants exist at all.

It sets the real engineering constraint of every model in modules 17-18, and it is the reason a long-context feature is a serious systems undertaking rather than a configuration change. Naming it plainly also makes the RNN comparison honest: recurrence is O(n · d²) and linear in n, so for very long sequences its cost profile is genuinely better — attention's win is parallelism, not total arithmetic.