Embeddings & Tokenization

A tokenizer maps text to integer token IDs, commonly with BPE or WordPiece subwords rather than whole words. Subwords make a finite vocabulary capable of representing rare names, inflections, code, and new compounds by breaking them into reusable pieces. An embedding table maps each ID to a learned vector; adding position information yields the ordered sequence of vectors presented to a Transformer. The embedding table is trained like any other parameter matrix: only rows for tokens appearing in a batch receive direct gradient updates.

Tokenization fixes the vocabulary, sequence length, cost, and even what text a model can express. Embeddings give tokens with similar training contexts an opportunity to become nearby vectors, but proximity is a learned association, not a guaranteed human semantic truth. A tokenizer mismatch at inference is therefore catastrophic: the same text becomes different IDs, and the model is asked to interpret a language it was not trained to read.

A token is the atomic unit a model operates on: one entry in a fixed, finite vocabulary, referred to everywhere downstream by its integer index. The model has no access to the string a token came from. Its entire knowledge of that token is one row of the embedding matrix, learned from every context in which that index appeared during training. A sequence, to the model, is a list of integers in [0, vocab_size) — nothing more.

Almost every confusing behaviour at the text boundary follows from this. A model cannot spell a word it only ever saw as one token, because it never saw the letters; it cannot count characters reliably for the same reason; and a leading space is a *different token* from the same word without one, which is why prompts sometimes behave differently for no visible reason. Once you accept that the model's world is integers, these stop being mysteries and become predictable consequences.

Word-level tokenization assigns one ID per whole word, which needs a huge vocabulary and still fails on anything unseen. Character-level tokenization needs a tiny vocabulary and never fails, but makes sequences several times longer and forces the model to rediscover words from letters. Subword tokenization — BPE, WordPiece, Unigram — sits between them: it learns a vocabulary of frequent character sequences, keeps common words whole, and splits rare ones into pieces it already knows.

Subword tokenization solves one specific problem, and naming it precisely is worth more than the taxonomy: out-of-vocabulary tokens. A word-level model meeting dexamethasone for the first time has exactly one option — emit <unk>, discarding the word entirely — and any text containing it becomes partly unreadable. A subword model splits it into dexa, meth, asone and produces a composed representation that is imperfect but genuinely informative. That is why subword won: it makes a *finite* vocabulary able to represent *unbounded* text.

nn.Embedding(V, d) holds a (V, d) weight matrix and, given integer indices, returns the corresponding rows. It is mathematically identical to one-hot encoding each index into a length-V vector and multiplying by the same matrix — the one-hot vector selects exactly one row. Computationally the two are not remotely the same: the matrix multiply materialises a (batch, seq, V) tensor and does batch × seq × V × d multiply-accumulates, almost all of them against zeros, while the lookup does a gather and touches only the rows it needs.

This is a real distinction worth stating, not pedantry. With V = 50,000 and d = 768, a batch of 8 sequences of 1024 tokens one-hot encoded is 400 million floats — around 1.6 GB — to compute something a gather does with 8192 row copies. The gradient side matters too: backward through the gather writes into only the rows that appeared in the batch, which is why an embedding table trains sparsely and why a token that never occurs in your fine-tuning data keeps its pretrained vector untouched.

embedding_dim is how many numbers describe each token. It is a capacity decision: too few dimensions and distinct tokens are forced to share directions in a cramped space, losing distinctions the model needed; too many and you pay for parameters that carry little extra information. The embedding table alone is V × d parameters, which for a large vocabulary is often a substantial fraction of a small model's total size.

It is one of the few architecture hyperparameters whose cost you can compute exactly before training anything, which makes it a good place to be deliberate rather than superstitious. A 50,000-token vocabulary at d = 128 is 6.4M parameters; at d = 1024 it is 51M. In a 100M-parameter model that is the difference between the embeddings being a rounding error and being half the model. Returns also flatten: most tasks show a clear knee beyond which extra dimensions buy very little held-out improvement.

Nothing in the architecture asks for this. It falls out of the objective: two tokens that appear in interchangeable contexts are asked to support the same predictions, so gradient descent pushes their rows toward whatever vector serves those predictions well. king and queen both frequently precede reigned and follow the, so both rows get nudged by similar gradients, and they converge. The similarity is a *consequence* of shared predictive role, not a property the model was told to encode.

It explains both the power and the limits. The power: an embedding table trained on enough text transfers useful structure to any downstream task, because the geometry already encodes distributional relationships nobody labelled. The limit: proximity means 'used in similar contexts', which is not the same as 'means a similar thing'. Antonyms are famously close — hot and cold occupy nearly identical syntactic slots — so an embedding distance is evidence about usage, never a semantic guarantee.

A tokenizer's granularity trades two costs against each other. A large vocabulary keeps more words whole, so the same document becomes fewer, longer tokens — shorter sequences, cheaper attention, but a bigger embedding and output matrix. A small vocabulary splits more aggressively: cheaper tables, longer sequences, and a quadratically more expensive attention step. The product that matters is not either number alone but total compute for the text you actually process.

The asymmetry decides it. Embedding and output-projection cost grows *linearly* with vocabulary size; attention cost grows *quadratically* with sequence length. So for long-context models the pressure runs toward larger vocabularies, because paying linearly to buy a shorter sequence is a good trade. It also has a practical consequence people meet without understanding: the same text costs a different number of tokens on two models, and non-English text often costs two to three times more on a tokenizer fitted mostly to English.

Token IDs are meaningful only relative to the tokenizer that produced them. Feed a model IDs from a different tokenizer and nothing errors: the indices are in range, the embedding lookup succeeds, the forward pass runs, and a fluent-looking but content-free output comes out. Every ID has silently been reinterpreted as a different token, so the model is reading a shuffled cipher of your text.

It is the single most damaging silent failure at this boundary, because every loud safety net is absent. There is no shape error, no dtype error, no NaN, no exception — just quietly wrong predictions. It happens in real systems whenever a tokenizer and a checkpoint are versioned separately: someone upgrades the model but the serving image caches the old vocabulary file, or a training pipeline builds its own vocabulary while inference uses the pretrained one. The fix is structural, not vigilant: ship the tokenizer *inside* the model artifact and assert the vocabulary hash at load time.