Sunday, 25 October 2026 · Weekend classic from 2020 · 6 min read
Language model loss follows predictable power laws with size, data, and compute
This paper finds that Transformer language model performance on cross-entropy loss scales as power laws with model size, dataset size, and training compute, enabling optimal compute allocation by training large models on modest data and stopping early.
Based on the paper Scaling Laws for Neural Language Models by Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown and 6 others
Understanding how language model performance depends on scale and compute
Language modeling is a core AI task where models predict the next token in text, and performance is measured by cross-entropy loss. Recent advances with Transformer architectures have shown strong results, but it is unclear how performance scales with model size, dataset size, and compute.
Determining how to allocate limited compute resources optimally between model size, dataset size, and training steps is challenging, especially since architectural choices like depth or width might affect performance.
This paper studies empirical scaling laws for Transformer language models trained on a large web text dataset, analyzing how loss depends on model parameters, dataset tokens, and compute, spanning many orders of magnitude.
Sign in to read the whole paper
You have read the opening. The rest is free with a Bezen account, and no card is needed.
In the rest of this paper
- How the method works, step by step, with a flow chart
- Three key formulas from the paper, explained symbol by symbol
- Three figures from the paper, and how to read them
- How it was tested, what was found, and where it falls short