STEPQuant reduces memory and preserves accuracy in low-bit quantization of recurrent states

Linear attention models replace the growing key-value (KV) cache of softmax attention with a fixed-size recurrent state matrix that summarizes past tokens, enabling constant memory usage per request regardless of sequence length. However, during serving, each concurrent request requires its own persistent recurrent state, causing memory usage to grow with concurrency and become a bottleneck.

Directly quantizing these recurrent states to low precision causes severe accuracy degradation because quantization errors accumulate and propagate through successive state updates. This error propagation is influenced by two factors: temporally, errors in long-lived memory persist across many decoding steps; spatially, errors in different key rows affect model outputs differently, and state magnitudes vary significantly across rows and columns.

Existing uniform quantization methods fail to address these temporal and spatial error patterns, motivating a structured quantization approach that accounts for when (temporal persistence) and where (spatial impact) errors matter in the recurrent state matrix.