Saturday, 14 November 2026 · Weekend classic from 2016 · 6 min read
Layer normalization stabilizes and speeds up training, especially in recurrent networks
Layer normalization normalizes summed inputs across neurons in a layer for each training case, enabling stable and faster training of deep and recurrent neural networks without batch size constraints, outperforming batch normalization in many tasks.
Based on the paper Layer Normalization by Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton
Challenges in training deep and recurrent neural networks with batch normalization
Training deep neural networks is computationally expensive and often requires many days of training. Batch normalization reduces training time by normalizing neuron inputs using statistics computed over mini-batches, but its effectiveness depends on batch size and it is difficult to apply to recurrent neural networks (RNNs) where sequence lengths vary.
Batch normalization requires maintaining running averages of mean and variance per layer, which is straightforward in feed-forward networks but problematic in RNNs due to varying sequence lengths and the need for separate statistics at each time step.
Moreover, batch normalization is not suitable for online learning or very small mini-batches, limiting its applicability in some scenarios. These challenges motivate alternative normalization methods that do not depend on batch statistics.
Sign in to read the whole paper
You have read the opening. The rest is free with a Bezen account, and no card is needed.
In the rest of this paper
- How the method works, step by step, with a flow chart
- Two key formulas from the paper, explained symbol by symbol
- Three figures from the paper, and how to read them
- How it was tested, what was found, and where it falls short