News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
RWKV-4

RWKV-4

A dense recurrent model with transformer-level performance

RWKV-4 is a large language model architecture that merges the training efficiency of transformers with the inference speed of recurrent neural networks. Released in 2023 by a broad research coalition, it scales to 14 billion parameters while maintaining linear memory usage.

Share on WhatsApp

  • Scales to 14 billion parameters
  • Combines transformer training with RNN inference
  • Linear memory and computational complexity
  • Largest dense RNN trained at release
  • Performance matches similarly sized transformers

What it is

RWKV-4 represents a specific iteration of the Receptance Weighted Key Value architecture. It is designed to address the computational bottlenecks found in standard transformer models. While transformers require memory and processing power that grow quadratically with sequence length, this model uses a recurrent approach. This allows it to scale efficiently. The project reached a size of 14 billion parameters, making it the largest dense recurrent neural network trained at the time of its publication. It aims to deliver performance comparable to similarly sized transformer models while using significantly less resources during inference.

How it came about

The model was introduced in a paper submitted to arXiv in May 2023. A large team of researchers collaborated on the project, including Bo Peng, Eric Alcaide, and Quentin Anthony among many others. The work was revised later in December 2023. The development focused on reinventing recurrent neural networks for the modern era of natural language processing. By combining the parallelizable training benefits of transformers with the efficient inference of recurrent networks, the team sought to reconcile long-standing trade-offs in sequence processing tasks.

Inside the model

The core innovation lies in its linear attention mechanism. This design allows the model to be formulated as either a transformer or a recurrent neural network. During training, it leverages parallel computation similar to transformers. During inference, it maintains constant computational and memory complexity, a trait typical of recurrent networks. This hybrid approach avoids the quadratic scaling issues of standard attention mechanisms. The architecture supports dense parameter usage, distinguishing it from sparse mixture-of-experts models. The training process enables the model to handle long sequences without the exponential increase in memory demand seen in traditional designs.

What it can do

The primary capability demonstrated is performance on par with similarly sized transformer models. Despite being a recurrent architecture, which historically struggled to match transformer accuracy, RWKV-4 achieves competitive results. It handles natural language processing tasks effectively. The model proves that recurrent structures can scale to billions of parameters without losing efficacy. Its efficiency makes it suitable for applications where memory and computational costs are critical constraints. The authors suggest that future work can build on this architecture to create even more efficient models for sequence processing.

How to use it

The model weights and associated code are available through open platforms such as Hugging Face. The research community has embraced the project, with tools and implementations shared via repositories like CatalyzeX and Replicate. The open nature of the release allows developers and researchers to experiment with the architecture. There are no indications of proprietary restrictions or API-only access in the provided materials. The collaborative spirit of the project encourages further development and sharing within the artificial intelligence community.

Why it matters

This work marks a significant step in reconciling the trade-offs between computational efficiency and model performance. It challenges the dominance of pure transformer architectures by showing that recurrent models can scale effectively. The success of RWKV-4 suggests that linear attention mechanisms offer a viable path for creating more efficient large language models. It influences the field by demonstrating that dense recurrent networks can compete with transformers in both size and capability, potentially reducing the environmental and financial costs of deploying large models.

Questions people ask about RWKV-4

How does RWKV-4 differ from standard transformers?

Standard transformers have memory and compute needs that grow quadratically with sequence length. RWKV-4 uses a linear attention mechanism and recurrent structure, keeping memory and computational complexity constant during inference while allowing parallel training.

What is the maximum size of the RWKV-4 model?

The model scales up to 14 billion parameters. This makes it the largest dense recurrent neural network ever trained at the time of its announcement, demonstrating that RNNs can scale to sizes previously dominated by transformers.

Is RWKV-4 available for public use?

Yes, the model is part of an open research effort. Code, data, and media associated with the article are available on platforms like Hugging Face and GitHub, allowing the community to access and build upon the architecture.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on RWKV-4 yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2305.13048. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.