Multi-head attention
Running several attention operations side by side, so different heads can track different kinds of relationship.
Instead of one large attention step, a transformer layer splits the work into several heads, typically from 8 to over 100, each with its own smaller queries, keys and values. One head might follow grammar, another track which noun a pronoun refers to, another copy repeated phrases. Their outputs are joined and mixed back together. Studying what individual heads do has been a starting point for interpretability research. Later variants share keys and values across heads to save memory.