DiffusionART stabilizes and speeds up reinforcement learning fine-tuning of diffusion models

Diffusion models generate images and videos by gradually denoising from noise to clean samples, but they are typically trained to match data distributions rather than optimize downstream criteria like visual or motion quality. Online reinforcement learning (RL) fine-tunes these models using black-box reward signals that evaluate the final generated outputs, which is crucial for improving generation quality in practice.

A key difficulty in online diffusion RL is credit assignment: the reward is only observed after a full output is generated, but the model must update many intermediate denoising states effectively. Existing methods split into reverse-process RL, which updates states visited during generation but suffers from poor credit assignment and computational overhead, and forward-process RL, which is more efficient but unstable without careful tuning of exponential moving average (EMA) off-policy training.

Forward-process methods like DiffusionNFT use positive and negative reward-weighted objectives but require joint training and EMA tuning to avoid collapse. Reverse-process methods like Flow-GRPO have better credit assignment but are computationally expensive and less efficient. Neither approach is both efficient and stable without per-task tuning, motivating a new method that addresses these limitations.