KV cache
Stored keys and values from earlier tokens, kept so the model does not recompute them for every new token.
Also: key-value cache
When a decoder writes a new token, attention needs the keys and values of every earlier token. Recomputing them each step would waste enormous effort, so they are saved in the KV cache. The cache grows with every token and every layer and, for long conversations, can take more memory than the model's weights. Grouped-query attention, multi-head latent attention, sliding windows and cache quantisation all shrink it. Prompt caching reuses the cache across requests that share the same opening text.