The encyclopedia · R&D & Science · Technical decision · 2015-2016
ResNet trained 152-layer networks by having each layer learn only the delta from its input
Microsoft's residual network let a model learn a small correction to its input, so depth stopped hurting and won ImageNet.
Microsoft Research · Microsoft
the move
Deeper neural networks should be stronger, but in practice they got harder to train and accurate networks degraded as depth grew, a problem called the degradation problem.
He, Zhang, Ren and Sun at Microsoft Research reformulated the layers to learn residual functions with reference to their input, via identity shortcut connections, instead of learning the unreferenced function directly.
On ImageNet, residual nets of up to 152 layers, eight times deeper than VGG with lower complexity, trained well and an ensemble reached a 3.57% error, winning first place at ILSVRC 2015.
why it works
- The identity shortcut gives gradients an easy path, so optimization doesn't collapse as depth grows.
- Learning the residual lets a useless layer be driven to zero, making extra depth harmless.
- Empirically, residual nets gain accuracy with more depth, which contradicts the degradation problem.
- The 152-layer result beat shallower networks, so depth became a lever rather than a risk.
what transfers
When a system can't scale because each added component risks hurting it, give every component a cheap no-op path so adding more is always at least harmless.
what came after
ResNet became a building block of modern computer vision and the basis for the ILSVRC and COCO 2015 wins, and the residual shortcut idea was carried into other architectures, including the later transformers.
references
- Deep Residual Learning for Image Recognition
- Deep Residual Learning for Image Recognition — CVPR 2016
spotted an error? The archive wants to know.