← Back to Basic Science

Layer Normalization

2016
Computer Science (theoretical)Machine Learning TheoryArchitecture Componentfoundational

Normalizes activations across the feature dimension within each training example rather than across the batch, removing batch-size dependence from normalization (Ba, Kiros & Hinton, "Layer Normalization," arXiv:1607.06450, 2016). Became the standard normalization sub-layer in sequence-transduction architectures, where batch normalization is awkward to apply.

Originators

  • Ba, J.L.
  • Kiros, J.R.
  • Hinton, G.E.

Landmark Paper

W3037932933 ↗
Not retracted (OpenAlex)

Checked 2026-09-19 — interim signal only, see docs/BASIC_ROADMAP.md Phase 10

Connections

  • is component of Transformer
    basis: reasoned

    Section 3.1 of Vaswani et al. 2017 follows every residual connection with layer normalization (ref [1], Ba, Kiros & Hinton), used instead of batch normalization because it is independent of batch size and applies uniformly across variable-length sequences.