Saturday, 24 October 2026 · Weekend classic from 2015 · 5 min read
Residual connections enable training of very deep neural networks with improved accuracy
This paper introduces residual learning, reformulating layers to learn residual functions with identity shortcuts, enabling training of networks up to 152 layers deep. On ImageNet, a 152-layer residual net achieves 3.57% top-5 error, winning ILSVRC 2015.
Based on the paper Deep Residual Learning for Image Recognition by Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
Challenges of training very deep neural networks
Deep convolutional neural networks have driven breakthroughs in image recognition by stacking many layers to learn hierarchical features. Increasing depth enriches representations but introduces optimization difficulties.
Although normalization and initialization methods have mitigated vanishing/exploding gradients, deeper networks still suffer from a degradation problem: adding layers beyond a certain depth leads to higher training error, not just overfitting.
This degradation suggests that deeper networks are harder to optimize, as the deeper model should at least match the performance of its shallower counterpart if identity mappings were easy to learn, but current solvers fail to find such solutions efficiently.
Sign in to read the whole paper
You have read the opening. The rest is free with a Bezen account, and no card is needed.
In the rest of this paper
- How the method works, step by step, with a flow chart
- The key formula from the paper, explained symbol by symbol
- Three figures from the paper, and how to read them
- How it was tested, what was found, and where it falls short