Pretraining & Fine-Tuning

Pretraining learns broad statistical structure from large, diverse corpora using a self-supervised objective such as next-token prediction. Fine-tuning then adapts that base model with curated labelled examples, instructions, or preference data so its outputs suit a narrower task or behaviour target. Parameter-efficient methods such as LoRA add small trainable low-rank updates while keeping most base weights frozen, reducing memory and making several adaptations practical to store. The data format, prompt template, target construction, and split policy are all part of the fine-tuning method—not preprocessing footnotes.

This division explains how capable models can be adapted without reproducing their original compute budget, and why data quality often matters more than a larger fine-tuning run. Fine-tuning can improve a task, but it can also cause catastrophic forgetting, narrow a model's style, or teach benchmark leakage; compare against the untouched base model on held-out and representative evaluations.

Modern practice splits training into two phases with almost nothing in common. Pretraining runs a self-supervised objective over an enormous, broad, unlabelled corpus for a long time at high learning rate, producing a general base model. Adaptation then runs a small, curated, usually labelled dataset for a short time at a much lower learning rate, producing a model suited to one task or behaviour. The phases differ in data volume by five or six orders of magnitude, and in compute by four or five.

Stating it plainly matters because the two phases have opposite failure modes and opposite instincts. In pretraining, more data is almost always better and overfitting is barely a concern; a run underfits and you scale it up. In fine-tuning, more data is rarely the constraint, overfitting arrives within an epoch or two, and the winning move is usually *fewer, better* examples and an earlier stop. Someone who carries pretraining intuitions into a fine-tuning run — train longer, raise the learning rate, add data — reliably destroys a good base model.

The pretraining objective is not the task anyone cares about. Nobody needs a next-token predictor. What makes it valuable is that predicting the next token well *requires* internal machinery that is useful for almost everything else: representations that track syntax, entity identity, factual associations, numeric relations and discourse state. Pretraining is therefore a way of buying general representations with cheap, abundant, unlabelled data instead of expensive, scarce labels.

This reframes the label economy, which is the actual reason the paradigm took over. Labelled data is expensive and domain-specific; raw text and images are effectively free and general. A self-supervised objective converts the free resource into the expensive one, so the scarce labels only have to teach the last, narrow step — how to format an answer, which of the model's existing distinctions matters here — rather than how language works. That is why a hundred well-chosen fine-tuning examples can now outperform a bespoke model trained on fifty thousand labels from scratch.

Full fine-tuning updates every weight. LoRA freezes the original weight matrix W entirely and learns a low-rank update instead: two thin matrices A (r × k) and B (d × r) whose product is added at inference, so the effective weight is W + BA. With r far smaller than d and k, the trainable parameter count collapses — a 4096 × 4096 matrix has 16.8M weights, but a rank-8 update has only 65,536, a factor of 256. Adapters take a related approach with small inserted bottleneck layers.

The memory saving is much larger than the parameter saving, and that is the practical point. Optimiser state for AdamW is roughly two extra float32 values per *trainable* parameter, and gradients one more, so full fine-tuning of a 7B model needs tens of gigabytes beyond the weights themselves. LoRA at rank 8 makes that term nearly disappear, which is what moves fine-tuning a large model from a multi-GPU job to a single-GPU one. It also changes the storage story: a LoRA adapter is a few megabytes, so one frozen base model can serve dozens of task-specific adaptations that are swapped at request time rather than deployed as separate multi-gigabyte checkpoints.

A raw pretrained language model continues text; it does not answer questions, because nothing in next-token prediction distinguishes a request from a document to be extended. Instruction tuning fixes this with supervised fine-tuning on (instruction, good response) pairs, teaching the model that a prompt in that format should be *followed*. RLHF goes further: humans rank alternative responses, those rankings train a reward model, and the language model is then optimised to score well under it — learning from comparative preference rather than from a single demonstrated answer.

The shape of the idea is what matters, and it is this: some qualities are much easier to *rank* than to *demonstrate*. Asking an annotator to write the ideal answer is hard, slow and inconsistent; asking them which of two answers is better is fast and fairly reliable. RLHF is a machine for converting that cheap comparative signal into a training gradient. It is also the stage responsible for most of what people perceive as a model's character — helpfulness, hedging, refusals, verbosity, format habits — none of which came from pretraining.

Fine-tuning moves weights to reduce loss on the new data, and nothing in that objective preserves anything else. With a large pretrained model the loss is already low everywhere, so the gradient from a narrow dataset is free to trade away capabilities that dataset never exercises. A model fine-tuned hard on medical summarisation can lose fluency in other languages, forget how to write code, and stop following instruction formats it handled perfectly before — while its held-out medical summarisation metric improves the whole time.

Module 14 raised forgetting for vision backbones; here the stakes and the detection problem are different. A vision backbone has one downstream task, so a regression shows up in that task's metric. A general language model has effectively unbounded capability surface, and your fine-tuning evaluation covers a vanishing fraction of it. So the regression is invisible by construction: every number on your dashboard goes up. The defences are structural — keep a fixed general-capability evaluation suite beside the task metric, prefer LoRA (which constrains how far the function can move), use low learning rates, stop early, and mix a little general data into the fine-tuning set.

Pretraining and adaptation sit at opposite ends of a very steep curve. A frontier pretraining run costs millions of dollars, occupies thousands of accelerators for weeks, and is performed by a handful of organisations a handful of times. A LoRA fine-tune costs single-digit dollars, occupies one GPU for an hour, and is performed by anyone. The ratio between them is roughly six orders of magnitude.

That ratio is the reason the two-stage paradigm reshaped the field, and it is an economic argument rather than a technical one. Before it, every serious model was trained from scratch, so capability was gated on having a large labelled dataset and a large compute budget — which meant very few people could build anything good. Pretraining amortises the expensive phase across everybody who ever adapts the result: the cost is paid once, by someone else, and what remains for you is small enough to iterate on in an afternoon. Capability stopped being something you trained and became something you started from.

A model with billions of parameters and a thousand fine-tuning examples can memorise the entire dataset almost immediately. Training loss falls toward zero within one or two epochs while validation loss turns upward, and because the starting model was already highly capable, the memorised version can still produce fluent, confident output — so nothing *looks* broken. What has happened is that the model has learned your examples rather than your task.

The instinct this defeats is the ordinary one: when a model underperforms, train it more. Here, more training is precisely the cause. The symptoms are specific and recognisable — outputs that reproduce phrasing from the fine-tuning set verbatim, sharply degraded behaviour on inputs slightly outside its distribution, and a validation curve that bottoms out before the first epoch ends. The practical protocol is to hold out a real validation split, evaluate several times per epoch rather than once, stop at the minimum, and treat two or three epochs as an upper bound rather than a starting point.