Bezen

Your daily digest

Research papers, made simple to understand

The AI and machine learning papers people are reading most, each one rewritten in plain words: the problem, the idea, what was found and why it matters.

Reinforcement Learning & Alignment6 papers, newest first

    • Reinforcement Learning & Alignment

    Waypoint-OPD improves RL policy transfer by gating shifts using intermediate checkpoints

    Waypoint-OPD uses an intermediate teacher checkpoint to gate the RL-induced policy shift transferred to student models, improving mathematical and code reasoning performance across multiple model sizes by up to 2.18 percentage points over prior methods.

    Paper: Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation · Bingxi Hou, Guochao Jiang, Guofeng Quan and 4 others

    654 upvotes on Hugging Face16 Oct 20267 min readRead the article

    • Reinforcement Learning & Alignment
    • Training & Efficiency

    Cross-teacher feedback reduces gradient conflicts in multi-teacher on-policy distillation

    Road MOPD improves multi-domain distillation by using auxiliary teachers to guide token-level credit assignment and prevent conflicting gradient updates, boosting performance by over 5 points in high-conflict domains and improving overall results in three-domain training.

    Paper: Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation · Shi-Yu Tian, Yong Liu, Ximan Liu and 10 others

    234 upvotes on Hugging Face15 Oct 20266 min readRead the article

    • Reinforcement Learning & Alignment
    • Image & Video Generation

    DiffusionART stabilizes and speeds up reinforcement learning fine-tuning of diffusion models

    DiffusionART improves efficiency and stability in online reinforcement learning fine-tuning of diffusion models by aggregating reward signals before updating states and using a drift-adaptive restoration update, achieving up to 12x faster training and higher rewards without per-task tuning.

    Paper: DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models · Zhenglin Wan, Jiayu Chen, Zhengfeiyang Zhang and 6 others

    230 upvotes on Hugging Face12 Oct 20268 min readRead the article

    • Reinforcement Learning & Alignment
    • Training & Efficiency

    LoGRA cuts LLM RL training memory up to 45.7% using low-rank gradient sketches

    LoGRA compresses gradients into low-rank sketches during RL post-training of large language models, reducing memory use by up to 45.7% without hurting accuracy and enabling stable training of a 27B-parameter model on eight GPUs where dense Adam fails.

    Paper: LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketches · Shaokun Zhang, Yifan Zhang, Jian Hu and 5 others

    147 upvotes on Hugging Face11 Oct 20267 min readRead the article

    • AI Agents
    • Reinforcement Learning & Alignment

    Self-Retrospection Distillation teaches agents to anticipate task demands before acting

    This paper introduces Self-Retrospection Distillation (SRD), a method that uses post-hoc trajectory hindsight to train agents to predict task-relevant knowledge and pitfalls before interaction, improving performance especially when reward signals lack contrast.

    Paper: Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresight · Haoxiang Zhang, Qinglin Chen, Hiroaki Hayashi and 9 others

    146 upvotes on Hugging Face10 Oct 20266 min readRead the article

    • Reinforcement Learning & Alignment
    • Training & Efficiency

    TRACE reduces train-rollout quantization mismatch for efficient FP4 RL training of MoE LLMs

    TRACE aligns FP4 quantization between training and rollout in reinforcement learning of Mixture-of-Experts language models, achieving BF16-level performance with up to 5.4× rollout speedup and reduced memory overhead.

    Paper: TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Models · Xin Wang, Hao Yu, Zhengyang Zhuge and 9 others

    75 upvotes on Hugging Face7 Oct 20266 min readRead the article