Sunday, 11 October 2026 · 7 min read
LoGRA cuts LLM RL training memory up to 45.7% using low-rank gradient sketches
LoGRA compresses gradients into low-rank sketches during RL post-training of large language models, reducing memory use by up to 45.7% without hurting accuracy and enabling stable training of a 27B-parameter model on eight GPUs where dense Adam fails.
Based on the paper LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketches by Shaokun Zhang, Yifan Zhang, Jian Hu, Yueying Li and 4 others
Memory demands limit reinforcement learning for large language models
Reinforcement learning (RL) enhances large language models (LLMs) by improving them through feedback on generated responses, but it requires storing large gradients and optimizer states, which can consume multiple times more memory than the model weights themselves.
For example, a 7B-parameter model’s weights may require 14 GB, but Adam optimizer states add 56 GB, making RL training infeasible on many hardware setups despite the model fitting in memory.
Existing solutions include system-level optimizations like sharding and checkpointing, parameter-efficient methods like LoRA that freeze weights and train adapters, and gradient compression techniques that keep weights trainable but compress gradients and optimizer states. However, these approaches still face challenges in scaling RL training efficiently.
Sign in to read the whole paper
You have read the opening. The rest is free with a Bezen account, and no card is needed.
In the rest of this paper
- How the method works, step by step, with a flow chart
- Three key formulas from the paper, explained symbol by symbol
- Three figures from the paper, and how to read them
- How it was tested, what was found, and where it falls short