Sunday, 25 October 2026 · Weekend classic from 2022 · 6 min read
Fine-tuning GPT-3 with human feedback improves instruction following and truthfulness
InstructGPT models, fine-tuned from GPT-3 using human demonstrations and preference rankings via reinforcement learning, outperform much larger GPT-3 models in following instructions, truthfulness, and toxicity reduction on diverse tasks.
Based on the paper Training language models to follow instructions with human feedback by Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida and 16 others
Challenges in aligning large language models with user intent
Large language models trained to predict next tokens on internet text often produce outputs that are untruthful, toxic, or fail to follow user instructions, reflecting a misalignment between their training objective and user intent.
This misalignment leads to unintended behaviors such as fabricating facts, generating biased or harmful content, and ignoring explicit instructions, which is problematic for deployed applications relying on these models.
Aligning models to be helpful, honest, and harmless requires methods beyond scaling model size, as larger models do not inherently improve alignment or instruction-following capabilities.
Sign in to read the whole paper
You have read the opening. The rest is free with a Bezen account, and no card is needed.
In the rest of this paper
- How the method works, step by step, with a flow chart
- The key formula from the paper, explained symbol by symbol
- Two figures from the paper, and how to read them
- How it was tested, what was found, and where it falls short