Multi-head latent attention
DeepSeek's attention design that compresses keys and values into a small shared vector to save memory.
Also: MLA
Multi-head latent attention was introduced with DeepSeek-V2 in 2024 and used again in DeepSeek-V3 and R1. Rather than storing full keys and values for every head, each token stores one compressed latent vector, from which the keys and values are reconstructed when needed. This cuts the size of the KV cache sharply, letting the model serve long contexts and many users with less memory. It is one of several engineering choices that let DeepSeek train and run very large mixture-of-experts models at relatively low cost.