Waypoint-OPD improves RL policy transfer by gating shifts using intermediate checkpoints

Reinforcement learning (RL) can improve language models by tuning them with rewards derived from human preferences or verifiable criteria, such as in mathematical and code reasoning tasks. However, training large models with RL is expensive, so transferring the improvements from a teacher model to student models is desirable.

Standard on-policy distillation (OPD) trains a student to imitate the teacher’s token distributions on student-generated trajectories, but this ties supervision to the teacher’s absolute policy and its limitations. Direct On-Policy Distillation (Direct-OPD) transfers the teacher’s RL-induced policy shift by using the log-probability ratio between pre-RL and post-RL teacher checkpoints as a dense reward, but it only considers the endpoint difference.

The endpoint policy shift alone cannot reveal whether local action preferences evolved consistently during teacher training. The same endpoint shift could result from early and late shifts that either reinforce or partially cancel each other, making it unclear which local changes should guide the student.