Grouped-query attention
A form of attention in which several query heads share one set of keys and values, shrinking the KV cache.
Also: GQA, multi-query attention, MQA
In standard multi-head attention every head stores its own keys and values, which makes the KV cache large for long texts. Noam Shazeer proposed multi-query attention in 2019, in which all heads share a single key and value. Grouped-query attention, described by Google researchers in 2023, sits in between: heads are split into groups, and each group shares keys and values. Llama 2 70B, Llama 3, Mistral 7B and many other models use it, cutting memory use with little loss in quality.