Wednesday, 28 October 2026 · 6 min read
Multiscale Policy Optimization improves credit assignment for long-horizon LLM agents
This paper introduces Multiscale Policy Optimization (MPO), which aligns credit assignment and policy updates at the turn level while refining token-level credit, achieving consistent gains over strong RL baselines on search, tool-use, and automation tasks with 4B and 30B models.
Based on the paper Multiscale Policy Optimization for Long-Horizon LLM Agents by Tao Ren, Hui Yang, Jia Deng, Tingyang Chen and 15 others
Challenges in credit assignment for long-horizon LLM agents
Large language model (LLM) agents interact with environments by producing complete assistant turns (e.g., search queries, tool calls) token by token. The final task reward is only observed at the end of the entire interaction, making credit assignment difficult.
Two scales of credit assignment arise: assigning credit across turns (macro actions) and within turns across tokens. Existing RL methods either assign credit at the sequence level without distinguishing turns or tokens, or at the token level without explicitly separating turns, leaving a gap in capturing both scales.
Critic values learned by the agent show that most variance occurs across turns rather than within turns, motivating a policy optimization method that aligns credit assignment and policy updates at the turn scale while refining token-level credit within turns.
Sign in to read the whole paper
You have read the opening. The rest is free with a Bezen account, and no card is needed.
In the rest of this paper
- How the method works, step by step, with a flow chart
- The key formula from the paper, explained symbol by symbol
- Three figures from the paper, and how to read them
- How it was tested, what was found, and where it falls short