News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
RecurrentGemma

RecurrentGemma

Open language models using Google's Griffin recurrent-plus-attention architecture

RecurrentGemma is a family of open language models built on Google's Griffin architecture, which mixes linear recurrences with local attention. Released in April 2024 in 2B and 9B parameter sizes, with pre-trained and instruction-tuned variants, it reaches performance comparable to similar-sized Gemma models while training on fewer tokens.

Share on WhatsApp

  • Uses Google's Griffin architecture: linear recurrences plus local attention
  • Released in 2B and 9B parameter sizes, pre-trained and instruction-tuned
  • Fixed-size state cuts memory use and speeds inference on long sequences
  • Matches similarly sized Gemma baselines with fewer training tokens
  • Paper first posted 11 April 2024, revised 28 August 2024

What it is

RecurrentGemma is a family of open language models from Google, described in a paper released in April 2024. It comes in two sizes, 2B and 9B parameters, and each size has both a pre-trained and an instruction-tuned version. The models are built on Griffin, an architecture that combines linear recurrences with local attention instead of relying on attention alone. The design keeps a fixed-size internal state, which lowers memory use and makes generation efficient on long sequences. The authors report results comparable to Gemma models of similar size, even though RecurrentGemma saw fewer training tokens.

How it came about

The work was presented in a paper posted on 11 April 2024, with a revised version in August 2024. The author list runs to 62 researchers, led by Aleksandar Botev and Soham De, and includes Samuel L Smith, Razvan Pascanu, Laurent Sifre, Zoubin Ghahramani, Koray Kavukcuoglu, Demis Hassabis and Raia Hadsell among others. The models were released alongside the paper as open language models in two sizes.

Inside the model

Griffin, the architecture underneath RecurrentGemma, mixes two mechanisms. Linear recurrences carry information forward in a fixed-size state, so memory does not grow with sequence length. Local attention looks back over a limited window of recent tokens to capture nearby context. The authors say this combination performs well on language while keeping inference efficient, especially for long inputs. RecurrentGemma was trained on fewer tokens than the Gemma baselines it was compared against, yet the paper reports performance at a similar level to those models.

What it can do

The paper reports that RecurrentGemma reaches performance comparable to Gemma baselines of similar size, despite being trained on fewer tokens. Its fixed-size state reduces memory use and supports efficient inference over long sequences. The abstract does not give individual benchmark scores.

How to use it

RecurrentGemma is described as a family of open language models. Both the 2B and 9B sizes were made available in pre-trained and instruction-tuned variants.

Why it matters

RecurrentGemma is presented as a step beyond the standard transformer design for open language models, showing that a hybrid of recurrence and local attention can match similarly sized transformer models while using fewer training tokens and less memory at inference time.

Questions people ask about RecurrentGemma

What is RecurrentGemma?

It is a family of open language models from Google, released in April 2024. Two sizes were published, 2B and 9B parameters, each with a pre-trained and an instruction-tuned variant. They are built on the Griffin architecture rather than a standard transformer.

How is Griffin different from a normal transformer?

Griffin combines linear recurrences with local attention. The recurrence keeps a fixed-size state, so memory use does not grow with sequence length, while local attention covers recent tokens. The authors say this gives strong language performance and efficient inference on long sequences.

How does it compare to Gemma?

The paper reports performance comparable to Gemma baselines of similar size, and notes that RecurrentGemma reached that level while being trained on fewer tokens. No individual benchmark scores are given in the abstract.

Who wrote the work?

The paper lists 62 authors, starting with Aleksandar Botev and Soham De. Names on the list include Samuel L Smith, Razvan Pascanu, Laurent Sifre, Zoubin Ghahramani, Koray Kavukcuoglu, Demis Hassabis and Raia Hadsell.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on RecurrentGemma yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2404.07839. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.