Thursday, 15 October 2026 · 6 min read
Cross-teacher feedback reduces gradient conflicts in multi-teacher on-policy distillation
Road MOPD improves multi-domain distillation by using auxiliary teachers to guide token-level credit assignment and prevent conflicting gradient updates, boosting performance by over 5 points in high-conflict domains and improving overall results in three-domain training.
Based on the paper Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation by Shi-Yu Tian, Yong Liu, Ximan Liu, Tianrui Zhu and 9 others
Challenges in multi-teacher on-policy distillation with shared parameters
On-policy distillation (OPD) trains a student model by providing dense token-level supervision from a strong teacher on trajectories generated by the student’s current policy, improving credit assignment over reward-based RL methods.
Multi-teacher on-policy distillation (MOPD) extends OPD by routing domain-specific data to specialized teachers, each expert in a domain, and training a shared student model on all domains. However, updates from all domains act on the same shared parameters, which can cause conflicting gradient directions when teachers differ substantially in style or expertise.
This gradient conflict leads to asymmetric performance degradation, especially in disadvantaged domains, limiting the benefits of joint multi-domain training. Domain routing alone does not coordinate these conflicting updates, motivating methods to better integrate multi-teacher supervision.
Sign in to read the whole paper
You have read the opening. The rest is free with a Bezen account, and no card is needed.
In the rest of this paper
- How the method works, step by step, with a flow chart
- Three key formulas from the paper, explained symbol by symbol
- Two figures from the paper, and how to read them
- How it was tested, what was found, and where it falls short