Language Modeling

Autoregressive language models factor a sequence probability into next-token predictions: each token is predicted from the tokens to its left. Training shifts the target sequence by one position and applies a causal mask; generation repeatedly samples or selects one token, appends it to context, and runs the next step. Masked language models instead hide selected tokens and use context on both sides, which is useful for representation learning but does not directly define left-to-right generation. Both commonly use cross-entropy, yet their visible context and inference procedures are fundamentally different.

It explains why a decoder-only model generates left to right and why likelihood, perplexity, sampling temperature, and top-p are tied to the same token distribution. Sampling settings are a decision policy over logits: temperature rescales confidence, while top-p truncates the candidate set; neither makes an unsupported answer more truthful or a weakly trained distribution more knowledgeable.

A language model defines a probability distribution over sequences, factored by the chain rule into conditional next-token probabilities: P(x1..xn) = Π P(xi | x1..xi-1). Training maximises the log-likelihood of real text under that factorisation, which is exactly minimising cross-entropy between the model's predicted distribution and the one-hot true next token. There is no separate loss per task; there is one loss, applied at every position of every sequence.

The surprising part is how much this teaches. To predict the next token well across a large, diverse corpus, a model has to implicitly acquire syntax (to close a bracket), factual associations (to finish the capital of France is), arithmetic (to complete a sum), discourse structure (to keep a pronoun's referent straight), and style (to match a register). None of these were supervised targets; each is instrumentally useful for reducing next-token loss, so gradient descent builds whatever internal machinery serves them. This is why one cheap, label-free objective on enough text yields a model useful for tasks nobody enumerated in advance.

During training every position is conditioned on the *ground-truth* prefix, not on what the model itself would have predicted. That is teacher forcing, and it is what lets a whole sequence be scored in one parallel forward pass. At inference there is no ground truth: the model conditions on its own previous outputs, one token at a time. So training and generation run the model in two genuinely different regimes — parallel on correct history, sequential on self-generated history.

The mismatch is called exposure bias, and it is worth naming because it explains a real failure shape. A model has never been trained on a prefix containing its own mistake, so once one appears the input is off the distribution the model was fitted to, and errors compound rather than being corrected. It is also why training loss can look excellent while long generations drift, repeat, or collapse: the loss was never measured in the regime that generation actually runs in. Practically, it is the reason long-form generation quality has to be evaluated by generating, not by scoring.

Perplexity is the exponentiated mean negative log-likelihood per token: PPL = exp(mean cross-entropy in nats). Its unit is interpretable — a perplexity of k means the model was, on average, as uncertain as if it had been choosing uniformly among k equally likely tokens. A perfect model scores 1; a model that assigns uniform probability over a 50,000-token vocabulary scores 50,000.

It is the standard language-modelling metric because it is a monotone function of the training loss, so it compares models without inventing a task, and its scale is intuitive in a way that raw nats are not. But it is comparable only across models sharing a tokenizer and an evaluation corpus. A model whose tokenizer splits text more finely gets more, easier predictions and a flattering perplexity for identical text — which is why cross-tokenizer perplexity comparisons are close to meaningless and bits-per-byte is sometimes reported instead.

The model emits a distribution; a decoding strategy turns it into one token. Greedy decoding always takes the argmax. Temperature T divides the logits before softmax: T < 1 sharpens the distribution toward the mode, T > 1 flattens it. Top-k truncates to the k highest-probability tokens and renormalises. Top-p (nucleus) truncates to the smallest set whose cumulative probability reaches p, so the candidate set is large where the model is genuinely uncertain and small where it is confident.

Every one of these trades coherence against diversity, and the right point on that trade is a product decision rather than a modelling one. Greedy and low temperature give repeatable, safe, often flat output, and are what you want for extraction, classification or code completion. Higher temperature and a wide nucleus give variety at the cost of more off-topic or incoherent choices, and suit brainstorming or creative drafting. Top-p is usually preferred to top-k because a fixed k is too permissive after a confident prediction and too restrictive after an uncertain one — it ignores the shape of the distribution it is truncating.

A masked language model replaces a fraction of tokens (classically 15%) with a [MASK] placeholder and trains the model to recover them using context on *both* sides. There is no causal mask: every position sees the whole sequence. This makes each position's representation bidirectionally contextualised, which is excellent for understanding tasks, and makes left-to-right generation impossible, because the model was never trained to continue from a prefix alone.

The contrast with next-token prediction is the clarifying point, and it is a contrast of *supervision density* against *representation quality*. Next-token prediction supervises every position of every sequence, so it extracts maximal signal per token of data and yields a usable generator; masked modelling supervises only the masked 15%, so it is roughly six times less data-efficient per token, but each representation it learns has seen both directions. That is precisely why encoder models dominate classification and retrieval while decoder-only models dominate generation — and why the field's shift toward generative use turned next-token prediction into the default pretraining objective.

A model's context length is the maximum number of tokens it can attend over in one forward pass, fixed by its positional scheme and its training. Within that window, standard self-attention materialises an n × n score matrix per head per layer, so both memory and compute for the attention step grow with the square of n while the feed-forward cost grows only linearly. Beyond the window, a model has no mechanism at all — tokens outside it are not down-weighted, they are absent.

The quadratic term is what makes long context an engineering problem rather than a configuration value. Doubling the window quadruples attention cost, and at long lengths the attention term overtakes everything else in the budget, so context length is one of the most expensive dials in a deployment. It also changes system design: a retrieval step that selects 2,000 relevant tokens is often cheaper *and* more accurate than pushing 100,000 tokens through the window, because attention over mostly-irrelevant context both costs more and dilutes the signal.

Greedy decoding — and temperature close to zero, which converges to it — is deterministic, and deterministic decoding of a language model tends to fall into loops. Once the model emits a phrase, that phrase is now in the context, which makes the tokens that followed it before slightly more likely, which makes the loop slightly more likely, and the feedback runs away. The output is not random noise; it is a highly confident, perfectly fluent repetition of the same clause forever.

It is worth naming as a trap because the instinct when output looks unstable is to *lower* temperature, which makes this failure mode worse rather than better. There is also a real distinction people get wrong: temperature = 0 is not a valid softmax temperature — it divides by zero — so libraries special-case it into argmax. That means 'temperature 0' and 'temperature 0.01' are implemented by different code paths, and a request that works at 0 may behave differently at 0.01 for reasons nothing in the API documents. When you need determinism, ask for greedy decoding explicitly rather than approximating it with a tiny temperature.