Long context
The ability to read and use very long inputs, from hundreds of thousands to millions of tokens.
Also: context extension
Long-context models can take in whole books, hours of transcripts or entire code repositories at once. Builders usually pre-train on shorter texts and then extend the window in a later stage, rescaling rotary position embeddings and training on long documents. Efficient attention kernels, sliding windows, KV cache compression and hybrid state-space designs keep costs in check. Simple tests such as finding a planted fact are now easy for leading models, but harder tasks that combine many distant facts still show weaknesses.