Attention
The step in a transformer where each token weighs every other token and pulls in information from the ones that matter.
Also: attention mechanism
Attention lets a model decide which earlier words are relevant to the current one. Each token produces three vectors: a query, a key and a value. The query of one token is compared with the keys of all the others, the scores are turned into weights with softmax, and the token receives a weighted mix of their values. In "the cat sat because it was tired", attention helps "it" draw on "cat". Attention was first used in 2014 to improve machine translation and became the centre of the transformer in 2017.