Thursday, 12 November 2026 · 7 min read
SteerDistill improves quantized LLM agents by teacher-guided correction and distillation
SteerDistill uses a full-precision teacher to iteratively guide and correct a quantized LLM agent's failed actions, distilling these corrections for unassisted inference. It outperforms strong baselines on five benchmarks and reduces training cost by 67%.
Based on the paper SteerDistill: Quantizing LLM Agents via Closed-Loop Teacher Steering by Genming Cui, Xiangyu Li, Zixu Hao, Xin Ding and 2 others
Challenges in quantizing LLM agents without losing task success
Large language model (LLM) agents interact with environments over multiple turns to complete tasks, but repeated model calls make inference costly. Low-bit quantization reduces this cost but often degrades task success rates.
Examining failed rollouts of quantized agents reveals behavioral divergences: the agents overlook task requirements or environmental conditions, leading to environment transitions that differ from successful full-precision references. For example, an agent might try to read a file before entering its directory, violating tool constraints.
Correcting these divergences is difficult because offline training data may not cover failure contexts, and the student's own rollouts rarely produce correct responses in those contexts. In a diagnostic, unguided retries yield correct actions at only 9.6% of divergent turns, showing the challenge of self-correction.
Sign in to read the whole paper
You have read the opening. The rest is free with a Bezen account, and no card is needed.
In the rest of this paper
- How the method works, step by step, with a flow chart
- Three key formulas from the paper, explained symbol by symbol
- Three figures from the paper, and how to read them
- How it was tested, what was found, and where it falls short