Bezen
Daily papers

Wednesday, 7 October 2026 · 6 min read

Sign in to save for later

TRACE reduces train-rollout quantization mismatch for efficient FP4 RL training of MoE LLMs

TRACE aligns FP4 quantization between training and rollout in reinforcement learning of Mixture-of-Experts language models, achieving BF16-level performance with up to 5.4× rollout speedup and reduced memory overhead.

Based on the paper TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Models by Xin Wang, Hao Yu, Zhengyang Zhuge, Bochao Mao and 8 others

Challenges of low-precision rollout in RL for MoE language models

Reinforcement learning (RL) fine-tuning of large language models (LLMs), especially Mixture-of-Experts (MoE) models, requires generating many long token sequences (rollouts), which is computationally and memory intensive. To reduce this cost, low-precision rollout quantization, such as FP4 (4-bit floating point), is used to speed up generation and reduce memory footprint.

However, FP4 quantization introduces coarse numerical precision, causing discrepancies between the rollout execution path and the training execution path. This discrepancy can destabilize RL training by causing policy mismatch, especially in MoE models where small numerical differences can change expert routing decisions and amplify divergence.

Existing FP4 RL methods focus on improving quantization accuracy independently on training and rollout paths but do not directly minimize the difference between the two quantized paths. This leads to persistent train-rollout mismatch, degraded RL performance, or training collapse. Addressing this mismatch is critical for stable and efficient RL training with FP4 rollout.

Sign in to read the whole paper

You have read the opening. The rest is free with a Bezen account, and no card is needed.

Create a free accountLog in

In the rest of this paper

  • How the method works, step by step, with a flow chart
  • Three key formulas from the paper, explained symbol by symbol
  • Three figures from the paper, and how to read them
  • How it was tested, what was found, and where it falls short
TRACE reduces train-rollout quantization mismatch for efficient FP4 RL training of MoE LLMs · Bezen