News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
RuGPT-3

RuGPT-3

Russian language model based on GPT-2 architecture

ruGPT-3 is a pretrained transformer language model for Russian, developed by the SberDevices team. It was trained on 80 billion tokens and offers open weights for research and development purposes.

Share on WhatsApp

  • Trained on 80 billion Russian tokens
  • Developed by SberDevices NLP team
  • Based on GPT-2 transformer architecture
  • Final test perplexity of 13.6
  • Supports context length up to 2048 tokens

What it is

ruGPT-3 is a large language model designed specifically for processing and generating Russian text. The SberDevices team built it using a transformer architecture similar to GPT-2. It serves as a foundational tool for natural language processing tasks in Russian, supporting researchers and developers who need robust linguistic capabilities for this specific language.

How it came about

The model was developed by the NLP core team at SberDevices. Its training process and evaluation metrics are detailed in a preprint titled A Family of Pretrained Transformer Language Models for Russian. The project represents a significant effort to create high-quality pretrained models tailored to the Russian language ecosystem.

Inside the model

Training involved processing 80 billion tokens over three epochs with a sequence length of 1024. This phase took approximately 14 days using 128 graphics processing units. The team then fine-tuned the model for one additional epoch with an extended context window of 2048 tokens, which required a few more days on 16 GPUs. The final perplexity score on the test set was recorded at 13.6.

What it can do

The model demonstrates competence in Russian language tasks, achieving a perplexity of 13.6 on its test dataset. It supports context windows of up to 2048 tokens after fine-tuning, allowing it to handle moderately long sequences of text. Its performance is optimized for general Russian natural language understanding and generation.

How to use it

The model weights are available openly through platforms like Hugging Face. Developers can download the model for local use or integration into their own applications. The open nature of the release encourages further research and adaptation within the Russian speaking technical community.

Questions people ask about RuGPT-3

Who developed ruGPT-3?

The model was created by the NLP core team at SberDevices, a major Russian technology company. The team included researchers such as Dmitry Zmitrovich and Tatiana Shavrina.

What architecture does ruGPT-3 use?

It uses a transformer architecture based on GPT-2. This design is a dense decoder transformer, which is well-suited for generative language tasks.

How much data was used for training?

The model was trained on a corpus of 80 billion tokens. The initial training ran for three epochs, followed by one epoch of fine-tuning with longer sequences.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on RuGPT-3 yet, so the facts on this page come from the sources listed above: huggingface.co/ai-forever/rugpt3large_based_on_gpt2. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.