Model Compression

Pruning removes individual weights, channels, heads, or entire structures; structured pruning is usually more likely to produce real hardware speedups than merely sparse tensors. Quantization stores and computes values with fewer bits, either after training or with quantization-aware training that teaches the model to tolerate rounding. Distillation trains a smaller student to match a teacher's soft outputs or intermediate representations, transferring useful dark knowledge beyond hard labels. Compression targets memory, latency, throughput, energy, or cost—not an abstract parameter count alone.

A smaller checkpoint can still be slow if kernels are unsupported or runtime is dominated by sequence length and memory movement. Benchmark the deployed path on representative inputs, hardware, batch sizes, and concurrency—not a synthetic single request. Quality evaluation should include calibration, rare slices, output format compliance, and task-specific human judgments where a headline accuracy number hides unacceptable regressions.

Training optimises for final accuracy under a compute budget; serving has entirely different constraints. A model has to fit in the memory of whatever hardware you can afford to run it on, load fast enough that autoscaling and cold starts are tolerable, and answer within a latency budget set by the product rather than by the model. A checkpoint that trained beautifully on an 80 GB accelerator may need to run on a CPU, a phone, or a GPU instance whose hourly cost has to be justified per request.

This is the motivation for the whole module, and it is an economics argument. Inference cost is paid per request, forever; training cost is paid once. So a model served a hundred million times a month has an inference bill that dwarfs what it cost to train, and a 4× reduction in serving cost is worth far more than a 4× reduction in training cost. The latency side is a product constraint rather than an engineering preference: a search ranker that takes 800 ms is not a slow feature, it is a feature nobody uses. Compression is how a model that is accurate enough becomes a model that is deployable at all.

Quantization stores weights — and often activations — with fewer bits, typically int8 instead of float32. Each tensor gets a scale and a zero point that map its float range onto the integer range, so a value is recovered as scale * (q - zero_point). Post-training quantization applies this to a finished model, needing only a small calibration set to observe activation ranges. Quantization-aware training instead simulates the rounding during fine-tuning so the model learns weights that survive it, which recovers most of the accuracy lost by the post-training route.

It is the highest-value compression technique because it gives three benefits from one change: the model is four times smaller, four times less data moves per operation, and integer arithmetic units are faster than floating-point ones. For the many deep-learning operations that are memory-bandwidth-bound rather than compute-bound, the bandwidth saving alone is most of the speedup. The accuracy cost is usually small and, importantly, *predictable*: int8 post-training quantization typically costs well under a point of accuracy on vision and language models, and quantization-aware training reduces that further. Below int8 — int4 and lower — the losses become task-dependent and have to be measured rather than assumed.

Pruning removes parameters that contribute little, usually judged by magnitude. Unstructured pruning zeroes individual weights wherever they are, which can remove 80 to 90% of a network's weights with small accuracy loss. Structured pruning removes whole units — a channel, a filter, an attention head, a whole layer — which is much coarser and therefore costs more accuracy per parameter removed.

The distinction is the single most commonly missed point in compression, and it is about hardware rather than accuracy. An unstructured-pruned tensor is *the same shape* with zeros in it. A dense matrix multiply does exactly as much work as before, because the hardware does not know or care that some operands are zero — so 90% sparsity gives 0% speedup and 0% memory saving unless you have a sparse kernel and a runtime that supports it. Structured pruning removes a dimension, so the tensor is genuinely smaller and *every* piece of hardware runs it faster. This is why structured pruning dominates in practice despite being the less impressive technique on paper.

Distillation trains a small student model to reproduce a large teacher's *output distribution* rather than the ground-truth labels. The loss is typically a weighted sum: ordinary cross-entropy against the hard labels, plus a KL divergence against the teacher's softened probabilities at a raised temperature. Some variants also match intermediate representations or attention patterns.

It works because the soft distribution carries far more information per example than a one-hot label does. A label says 'this is a 7'; a teacher's distribution says 'this is a 7, and it looks somewhat like a 1, and not at all like an 8' — encoding a similarity structure over classes that the hard label discards entirely. That extra signal is why a student trained by distillation can outperform the identical architecture trained from scratch on the same data: it is learning from a richer target. Raising the temperature matters for the same reason — at T = 1 a confident teacher is nearly one-hot and the extra information is squeezed out of the tail, so the softening is what makes it visible.

A compression result is four numbers, not one: model size, p50 latency, p99 latency, and quality on a real evaluation set. Any technique can look good if you report only the number it improves. The comparison also has to be made on the deployment path — the same hardware, runtime, batch size and concurrency the service will actually use — because a technique that halves size and doubles latency on your target hardware has made things worse.

The four-column view is what makes the choice obvious, and it usually is obvious once the table exists. It also exposes the case people miss: int8 quantization is nearly always the first thing to try, because it improves size *and* latency for a fraction of a point of accuracy, while pruning and distillation cost more quality and require work. And it makes the combinations visible — distillation and quantization attack different things, so a distilled-then-quantized model is often the real answer rather than either alone.

Quantization, pruning and distillation look like three different techniques, and at the level of what they trade they are one. Each gives up a small, bounded amount of model quality to gain a large, measurable amount of size, latency or cost. Quantization gives up precision in the weights; pruning gives up the least-used parameters; distillation gives up the capacity of the larger architecture. In every case the exchange rate is heavily in your favour — a few tenths of a point for a factor of four.

Recognising the single shape is what makes the trade arguable rather than uncomfortable. The instinct in machine learning is that accuracy is the goal, so anything that reduces it is a regression to be resisted. In production the goal is user outcome per unit cost, and that calculus is different: a model 0.4 points less accurate that responds in 17 ms instead of 41 ms is better for almost any interactive product, because latency affects every user on every request while those 0.4 points affect a small fraction of predictions marginally. The trade is usually worth it — and saying *why* it is worth it, in terms of the product rather than the metric, is what gets it approved.

The failure that ends the module. A team quantizes or prunes a model, confirms the headline accuracy is within tolerance, and ships. What they did not check is that compression does not degrade uniformly: it disproportionately damages the rare, the long-tailed and the numerically delicate. Overall accuracy moves 0.3 points while the smallest class drops fifteen, calibration breaks so every confidence score is now wrong, or a structured output format the model used to emit reliably starts malforming under low precision.

The uniformity assumption is the bug. Compression removes capacity, and a model spends its capacity where the data is — so the parts removed first are the parts that served the least frequent inputs. An aggregate metric averaged over a distribution dominated by common cases is structurally incapable of showing this, which is why a compressed model can pass every dashboard check and fail exactly the users who are hardest to serve. The defence is a per-slice evaluation gate: the same evaluation you ran on the baseline, broken down by class, by segment and by calibration, with an explicit tolerance per slice rather than one tolerance on the mean.