← Back to Basic Science

Residual Connections

2015
Computer Science (theoretical)Machine Learning TheoryArchitecture Componentfoundational

Identity shortcut connections that let gradients skip layers directly, addressing the degradation problem in very deep networks and enabling training of substantially deeper models (He, Zhang, Ren & Sun, "Deep Residual Learning for Image Recognition," CVPR 2016). Originated in image recognition; reused structurally as a sub-layer wrapper in later sequence-transduction architectures.

Originators

  • He, K.
  • Zhang, X.
  • Ren, S.
  • Sun, J.

Landmark Paper

W2194775991 ↗
Not retracted (OpenAlex)

Checked 2026-09-19 — interim signal only, see docs/BASIC_ROADMAP.md Phase 10

Connections

  • is component of Transformer
    basis: reasoned

    Section 3.1 of Vaswani et al. 2017 wraps every sub-layer in a residual connection (ref [11], He et al.), "LayerNorm(x + Sublayer(x))", citing He et al.'s residual learning directly as the mechanism enabling the full 6-layer encoder/decoder stack to train.