News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM

State-space and recurrent

Mamba, RWKV and the models that avoid attention's cost on long texts. 12 models on EveryLLM so far.

State-space and recurrent models process text as a running summary rather than attending back over every earlier token. They carry a fixed-size state from one token to the next, so memory and time grow only in step with length, and generating each new token costs the same however long the text. The trade-off is recall: a fixed state can forget details that attention would look up exactly. Mamba and RWKV are the leading designs.

From S4 to Mamba

Structured state-space models, starting with S4 from Stanford in 2021, borrowed ideas from control theory to model long sequences efficiently. Mamba, published in December 2023 by Albert Gu of Carnegie Mellon and Tri Dao of Princeton, made the state depend on the input, letting the model choose what to keep, and matched transformers of similar size on language tasks. Mamba-2 followed in May 2024.

Recurrent revivals

RWKV, an open project led by Bo Peng and described in a 2023 paper, trains like a transformer but runs like a recurrent neural network. Other efforts include Microsoft's RetNet (2023) and xLSTM from Sepp Hochreiter's group in Linz (2024). In practice the largest releases mix these layers with some attention rather than dropping it: Nvidia's Nemotron-H models and IBM's Granite 4.0 (2025) use mostly Mamba-2 layers.

Famous state-space models

Further reading

The weekly digest

Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.