Daily papers

Sunday, 15 November 2026 · Weekend classic from 2023 · 7 min read

Direct Preference Optimization fine-tunes language models from human preferences without reinforcement learning

Direct Preference Optimization (DPO) trains language models to align with human preferences using a simple classification loss, matching or outperforming RLHF methods like PPO on sentiment, summarization, and dialogue tasks, while being more stable and efficient.

Based on the paper Direct Preference Optimization: Your Language Model is Secretly a Reward Model by Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon and 2 others

Challenges in aligning language models with human preferences

Large unsupervised language models learn broad knowledge but lack precise control over their outputs because their training is unsupervised and reflects diverse human data with varying goals and quality.

To steer models toward desired behaviors, methods collect human preference data comparing pairs of model outputs and fine-tune models to align with these preferences, often using reinforcement learning from human feedback (RLHF).

RLHF involves first training a reward model to predict human preferences, then using reinforcement learning to optimize the language model to maximize this reward while staying close to the original model, but this process is complex, unstable, and computationally expensive.

Sign in to read the whole paper

You have read the opening. The rest is free with a Bezen account, and no card is needed.

In the rest of this paper

  • How the method works, step by step, with a flow chart
  • The key formula from the paper, explained symbol by symbol
  • Three figures from the paper, and how to read them
  • How it was tested, what was found, and where it falls short
Direct Preference Optimization fine-tunes language models from human preferences without reinforcement learning · Bezen