Layer normalisation
A step that rescales each token's vector to a steady size inside every layer, keeping training stable.
Also: LayerNorm, RMSNorm
Layer normalisation, described by Jimmy Ba and colleagues in 2016, rescales the numbers in each hidden state so they have a consistent average and spread. This keeps values from growing or shrinking out of control across many layers. The original transformer normalised after each step. GPT-2 and most later models normalise before each step, which trains more reliably at scale. Many current models use RMSNorm, a simpler version from 2019 that skips subtracting the average and is slightly faster.