Sliding-window attention
Attention limited to a fixed number of recent tokens, so cost stops growing with the full length of the text.
Also: local attention
In sliding-window attention each token attends only to the last few thousand tokens rather than to everything before it. Information from further back can still travel indirectly, passed along from layer to layer. Mistral 7B used a 4,096-token window in 2023, and Gemma 2 and Gemma 3 alternate local sliding-window layers with layers of full global attention. The trade is lower memory and compute on long texts against a weaker direct view of distant passages.