RuGPT-3
Russian language model based on GPT-2 architecture
ruGPT-3 is a pretrained transformer language model for Russian, developed by the SberDevices team. It was trained on 80 billion tokens and offers open weights for research and development purposes.
- Trained on 80 billion Russian tokens
- Developed by SberDevices NLP team
- Based on GPT-2 transformer architecture
- Final test perplexity of 13.6
- Supports context length up to 2048 tokens
What it is
ruGPT-3 is a large language model designed specifically for processing and generating Russian text. The SberDevices team built it using a transformer architecture similar to GPT-2. It serves as a foundational tool for natural language processing tasks in Russian, supporting researchers and developers who need robust linguistic capabilities for this specific language.
How it came about
The model was developed by the NLP core team at SberDevices. Its training process and evaluation metrics are detailed in a preprint titled A Family of Pretrained Transformer Language Models for Russian. The project represents a significant effort to create high-quality pretrained models tailored to the Russian language ecosystem.
Inside the model
Training involved processing 80 billion tokens over three epochs with a sequence length of 1024. This phase took approximately 14 days using 128 graphics processing units. The team then fine-tuned the model for one additional epoch with an extended context window of 2048 tokens, which required a few more days on 16 GPUs. The final perplexity score on the test set was recorded at 13.6.
What it can do
The model demonstrates competence in Russian language tasks, achieving a perplexity of 13.6 on its test dataset. It supports context windows of up to 2048 tokens after fine-tuning, allowing it to handle moderately long sequences of text. Its performance is optimized for general Russian natural language understanding and generation.
How to use it
The model weights are available openly through platforms like Hugging Face. Developers can download the model for local use or integration into their own applications. The open nature of the release encourages further research and adaptation within the Russian speaking technical community.
Questions people ask about RuGPT-3
Who developed ruGPT-3?
The model was created by the NLP core team at SberDevices, a major Russian technology company. The team included researchers such as Dmitry Zmitrovich and Tatiana Shavrina.
What architecture does ruGPT-3 use?
It uses a transformer architecture based on GPT-2. This design is a dense decoder transformer, which is well-suited for generative language tasks.
How much data was used for training?
The model was trained on a corpus of 80 billion tokens. The initial training ran for three epochs, followed by one epoch of fine-tuning with longer sequences.
Good reading and sources
There is no English Wikipedia article on RuGPT-3 yet, so the facts on this page come from the sources listed above: huggingface.co/ai-forever/rugpt3large_based_on_gpt2. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.