Wednesday, 7 October 2026 · 7 min read
Compact strict alignment supervision outperforms broader coverage in cross-tokenizer distillation
This paper shows that in on-policy distillation between models with different tokenizers, focusing supervision on strictly aligned token positions with a small shared vocabulary subset retains most learning benefits, while adding broader span supervision reduces accuracy.
Based on the paper Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability by Bingxi Hou, Guochao Jiang, Guofeng Quan, Weiqing Li and 3 others
Challenges of cross-tokenizer on-policy distillation
On-Policy Distillation (OPD) trains a student language model on its own generated sequences using feedback from a teacher model. When the teacher and student use different tokenizers, their token boundaries and vocabularies differ, complicating direct comparison of predictions.
Cross-tokenizer OPD must align both the sequence positions and vocabulary entries between teacher and student. Strict 1:1 token alignments occur when a single student token corresponds exactly to a single teacher token covering the same text span. Mismatch groups involve multiple tokens on one or both sides to cover the same span.
Previous work aimed to increase alignment coverage by including mismatch groups and broader vocabulary matches, assuming more supervision signals improve learning. However, it is unclear whether this broader coverage provides reliable or helpful supervision.
Sign in to read the whole paper
You have read the opening. The rest is free with a Bezen account, and no card is needed.
In the rest of this paper
- How the method works, step by step, with a flow chart
- Three key formulas from the paper, explained symbol by symbol
- Three figures from the paper, and how to read them
- How it was tested, what was found, and where it falls short