News
EveryLLM

State-space and recurrent

Mamba, RWKV and the models that avoid attention's cost on long texts. 12 models on EveryLLM so far.

State-space and recurrent models process text as a running summary rather than attending back over every earlier token. They carry a fixed-size state from one token to the next, so memory and time grow only in step with length, and generating each new token costs the same however long the text. The trade-off is recall: a fixed state can forget details that attention would look up exactly. Mamba and RWKV are the leading designs.

From S4 to Mamba

Structured state-space models, starting with S4 from Stanford in 2021, borrowed ideas from control theory to model long sequences efficiently. Mamba, published in December 2023 by Albert Gu of Carnegie Mellon and Tri Dao of Princeton, made the state depend on the input, letting the model choose what to keep, and matched transformers of similar size on language tasks. Mamba-2 followed in May 2024.

Recurrent revivals

RWKV, an open project led by Bo Peng and described in a 2023 paper, trains like a transformer but runs like a recurrent neural network. Other efforts include Microsoft's RetNet (2023) and xLSTM from Sepp Hochreiter's group in Linz (2024). In practice the largest releases mix these layers with some attention rather than dropping it: Nvidia's Nemotron-H models and IBM's Granite 4.0 (2025) use mostly Mamba-2 layers.

Famous state-space models

Further reading

The weekly digest

Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.