RecurrentGemma
Open language models using Google's Griffin recurrent-plus-attention architecture
RecurrentGemma is a family of open language models built on Google's Griffin architecture, which mixes linear recurrences with local attention. Released in April 2024 in 2B and 9B parameter sizes, with pre-trained and instruction-tuned variants, it reaches performance comparable to similar-sized Gemma models while training on fewer tokens.
- Uses Google's Griffin architecture: linear recurrences plus local attention
- Released in 2B and 9B parameter sizes, pre-trained and instruction-tuned
- Fixed-size state cuts memory use and speeds inference on long sequences
- Matches similarly sized Gemma baselines with fewer training tokens
- Paper first posted 11 April 2024, revised 28 August 2024
What it is
RecurrentGemma is a family of open language models from Google, described in a paper released in April 2024. It comes in two sizes, 2B and 9B parameters, and each size has both a pre-trained and an instruction-tuned version. The models are built on Griffin, an architecture that combines linear recurrences with local attention instead of relying on attention alone. The design keeps a fixed-size internal state, which lowers memory use and makes generation efficient on long sequences. The authors report results comparable to Gemma models of similar size, even though RecurrentGemma saw fewer training tokens.
How it came about
The work was presented in a paper posted on 11 April 2024, with a revised version in August 2024. The author list runs to 62 researchers, led by Aleksandar Botev and Soham De, and includes Samuel L Smith, Razvan Pascanu, Laurent Sifre, Zoubin Ghahramani, Koray Kavukcuoglu, Demis Hassabis and Raia Hadsell among others. The models were released alongside the paper as open language models in two sizes.
Inside the model
Griffin, the architecture underneath RecurrentGemma, mixes two mechanisms. Linear recurrences carry information forward in a fixed-size state, so memory does not grow with sequence length. Local attention looks back over a limited window of recent tokens to capture nearby context. The authors say this combination performs well on language while keeping inference efficient, especially for long inputs. RecurrentGemma was trained on fewer tokens than the Gemma baselines it was compared against, yet the paper reports performance at a similar level to those models.
What it can do
The paper reports that RecurrentGemma reaches performance comparable to Gemma baselines of similar size, despite being trained on fewer tokens. Its fixed-size state reduces memory use and supports efficient inference over long sequences. The abstract does not give individual benchmark scores.
How to use it
RecurrentGemma is described as a family of open language models. Both the 2B and 9B sizes were made available in pre-trained and instruction-tuned variants.
Why it matters
RecurrentGemma is presented as a step beyond the standard transformer design for open language models, showing that a hybrid of recurrence and local attention can match similarly sized transformer models while using fewer training tokens and less memory at inference time.
Questions people ask about RecurrentGemma
What is RecurrentGemma?
It is a family of open language models from Google, released in April 2024. Two sizes were published, 2B and 9B parameters, each with a pre-trained and an instruction-tuned variant. They are built on the Griffin architecture rather than a standard transformer.
How is Griffin different from a normal transformer?
Griffin combines linear recurrences with local attention. The recurrence keeps a fixed-size state, so memory use does not grow with sequence length, while local attention covers recent tokens. The authors say this gives strong language performance and efficient inference on long sequences.
How does it compare to Gemma?
The paper reports performance comparable to Gemma baselines of similar size, and notes that RecurrentGemma reached that level while being trained on fewer tokens. No individual benchmark scores are given in the abstract.
Who wrote the work?
The paper lists 62 authors, starting with Aleksandar Botev and Soham De. Names on the list include Samuel L Smith, Razvan Pascanu, Laurent Sifre, Zoubin Ghahramani, Koray Kavukcuoglu, Demis Hassabis and Raia Hadsell.
Good reading and sources
There is no English Wikipedia article on RecurrentGemma yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2404.07839. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.