案例库 · 研发与科研 · 技术决策 · 2015-2016
这条还没译成中文，下面是英文原文。
ResNet trained 152-layer networks by having each layer learn only the delta from its input
Microsoft's residual network let a model learn a small correction to its input, so depth stopped hurting and won ImageNet.
Microsoft Research · Microsoft
那一手
Deeper neural networks should be stronger, but in practice they got harder to train and accurate networks degraded as depth grew, a problem called the degradation problem.
He, Zhang, Ren and Sun at Microsoft Research reformulated the layers to learn residual functions with reference to their input, via identity shortcut connections, instead of learning the unreferenced function directly.
On ImageNet, residual nets of up to 152 layers, eight times deeper than VGG with lower complexity, trained well and an ensemble reached a 3.57% error, winning first place at ILSVRC 2015.
为什么管用
- The identity shortcut gives gradients an easy path, so optimization doesn't collapse as depth grows.
- Learning the residual lets a useless layer be driven to zero, making extra depth harmless.
- Empirically, residual nets gain accuracy with more depth, which contradicts the degradation problem.
- The 152-layer result beat shallower networks, so depth became a lever rather than a risk.
可以搬走什么
When a system can't scale because each added component risks hurting it, give every component a cheap no-op path so adding more is always at least harmless.
后来呢
ResNet became a building block of modern computer vision and the basis for the ILSVRC and COCO 2015 wins, and the residual shortcut idea was carried into other architectures, including the later transformers.
资料来源
- Deep Residual Learning for Image Recognition
- Deep Residual Learning for Image Recognition — CVPR 2016
发现哪里写错了？告诉我们。