News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
RWKV-7 Goose

RWKV-7 Goose

Recurrent language model with constant memory usage

RWKV-7 Goose is a sequence modelling architecture released in March 2025 by Bo Peng and colleagues. It uses 2.9 billion parameters to match top multilingual performance while maintaining constant inference time per token.

Share on WhatsApp

  • Constant memory usage and inference time per token
  • 2.9 billion parameters match 3B SoTA on multilingual tasks
  • Trained on 3.1 trillion token open source corpus
  • Recognizes all regular languages and performs state tracking
  • Released under Apache 2.0 License with open code

What it is

RWKV-7 Goose is a new sequence modelling architecture designed for efficient processing. It operates with constant memory usage and constant inference time for each token. The largest version contains 2.9 billion parameters. Despite training on fewer tokens than many leading models, it achieves state of the art results for its size class on multilingual tasks. It matches current benchmarks for English language performance at the three billion parameter scale. The system is built to handle complex state tracking while remaining parallelisable during the training phase.

How it came about

The model was presented in a paper submitted to arXiv in March 2025. The work was led by Bo Peng along with seventeen other authors including Ruichong Zhang and Daniel Goldstein. The team developed four distinct models ranging from 0.19 billion to 2.9 billion parameters. These were trained on an extended open source corpus containing 3.1 trillion tokens. The release aimed to foster openness and allow for reproduction of the results by the wider community.

Inside the model

The architecture introduces a generalized formulation of the delta rule. This includes vector-valued gating and in-context learning rates. It also employs a relaxed value replacement rule. These changes allow the model to perform state tracking and recognize all regular languages. This capability exceeds the theoretical limits of standard Transformers under certain complexity conjectures. The design retains the ability to parallelize training, which is often difficult for recurrent models. The structure ensures that memory and inference time do not grow with sequence length.

What it can do

The 2.9 billion parameter model sets a new benchmark for three billion parameter systems on multilingual tasks. It performs comparably to the best existing models in English downstream evaluations. The architecture demonstrates strong capabilities in state tracking. It can recognize all regular languages, a task that challenges standard Transformer models. The performance is achieved despite being trained on a significantly smaller volume of data than some competitors.

How to use it

The creators released the models under the Apache 2.0 Licence. This permits broad use and modification. The dataset component listing is also available publicly. Training and inference code have been shared to support adoption and reproduction. The resources are hosted on platforms such as Hugging Face to facilitate community access.

Why it matters

The work challenges the dominance of Transformer architectures by showing that recurrent styles can achieve high performance. It provides evidence that constant memory models can handle complex linguistic tasks. The release of the large multilingual corpus supports further research in efficient sequence modelling.

Questions people ask about RWKV-7 Goose

What makes RWKV-7 different from Transformers?

It uses constant memory and inference time per token, unlike Transformers which scale with sequence length. It can recognize all regular languages and perform state tracking, exceeding standard Transformer limits under certain complexity conjectures.

How large is the RWKV-7 Goose model?

The largest model in the RWKV-7 series has 2.9 billion parameters. The researchers also trained smaller versions ranging down to 0.19 billion parameters on the same dataset.

Is the code and data available for public use?

Yes. The models, dataset component listing, and training and inference code are all released under the Apache 2.0 License. This allows for open reproduction and adoption by the community.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on RWKV-7 Goose yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2503.14456. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.