Positional encoding
Information added to token embeddings so the model knows the order of words, which attention alone ignores.
Also: position embedding
Attention treats its inputs as a set: without help, "dog bites man" and "man bites dog" would look the same. Positional encodings fix this. The original transformer added fixed sine and cosine waves of different frequencies to each embedding. GPT-2 and BERT learned a separate vector for each position, which limited them to the lengths seen in training. Most models since 2022 use rotary position embeddings or ALiBi, which encode the distance between tokens and stretch more gracefully to longer texts.