News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
RoBERTa

RoBERTa

A robustly optimised BERT pretraining study with released models and code

RoBERTa is a masked language model released in 2019 as a careful replication study of BERT pretraining. Yinhan Liu and colleagues at Facebook AI and the University of Washington found BERT was significantly undertrained and that hyperparameter choices mattered greatly. Their best model matched or exceeded later models on GLUE, RACE and SQuAD.

Share on WhatsApp

  • Replication study of BERT pretraining, published July 2019
  • Found BERT was significantly undertrained
  • Trained on 160GB of text from five corpora
  • Pretraining used 1024 V100 GPUs for 500K steps
  • Best model reported state-of-the-art results on GLUE, RACE and SQuAD

What it is

RoBERTa is a pretrained transformer encoder for English, trained to fill in hidden words in a sentence. It came out in July 2019 from a team led by Yinhan Liu, working across Facebook AI and the University of Washington. The work is a replication study of BERT: the authors re-ran BERT-style pretraining while measuring how much individual hyperparameters and the amount of training data change the outcome. Their conclusion was that BERT had been significantly undertrained, and that with better tuning it could match or beat every model published after it. The largest release, RoBERTa large, is meant to be fine-tuned on labelled tasks rather than used raw.

How it came about

The paper appeared on arXiv on 26 July 2019, listing ten authors including Myle Ott, Naman Goyal, Danqi Chen, Mike Lewis, Luke Zettlemoyer and Veselin Stoyanov. It set out to settle a problem in the field: pretraining is costly, datasets are often private and of varying sizes, and hyperparameter choices shift results enough to make fair comparison hard. The team released both models and code, first through the fairseq repository at Facebook's PyTorch project. The model card on Hugging Face was written by others, since the original team did not publish one.

Inside the model

RoBERTa learns from raw text with no human labels, using masked language modeling: 15% of tokens in an input are hidden and the model must recover them. Unlike BERT, the masking is redone every epoch instead of being fixed once. Byte-level Byte-Pair Encoding builds a 50,000-token vocabulary, and inputs are runs of 512 tokens that can cross document boundaries. Training data combined BookCorpus, English Wikipedia, CC-News, OpenWebText and Stories, about 160GB in total. Pretraining ran on 1024 V100 GPUs for 500,000 steps with batch size 8K, using Adam at learning rate 4e-4, weight decay 0.01 and 30,000 warmup steps.

What it can do

The paper reports state-of-the-art results on GLUE, RACE and SQuAD for its best model. Fine-tuned RoBERTa large scores listed on the model card include 90.2 on MNLI, 92.2 on QQP, 94.7 on QNLI, 96.4 on SST-2, 68.0 on CoLA, 96.4 on STS-B, 90.9 on MRPC and 86.6 on RTE. The model is case-sensitive, so 'english' and 'English' are treated differently. It is built for whole-sentence decisions such as classification and question answering; for text generation the card points users to GPT2 instead.

How to use it

Weights and code were released publicly, first in the fairseq examples and later on the Hugging Face model hub as FacebookAI/roberta-large. The model can be run directly for masked language modeling or used to pull features from text in PyTorch and TensorFlow. Fine-tuned versions for specific tasks are collected on the hub.

Safety and testing

The model card warns that training data drew on unfiltered internet text, which is not neutral, so predictions can carry bias. Its example shows gendered job associations in fill-mask output, and notes that such bias passes into any fine-tuned version of the model.

Reception and debate

The study's central claim was that BERT had been undertrained and that many reported gains since came from tuning choices rather than new ideas. The authors said this raised questions about where recent improvements actually came from. The paper reports its best model reaching state-of-the-art results on GLUE, RACE and SQuAD.

Why it matters

By releasing models and code alongside a careful measurement of pretraining choices, RoBERTa gave later work a stronger BERT baseline and a checklist of settings that matter, such as data volume, batch size and masking. Its finding that an older method, properly tuned, could hold its own against newer ones shaped how the field reads pretraining comparisons.

Questions people ask about RoBERTa

What is RoBERTa?

It is a masked language model for English, released in 2019 by researchers at Facebook AI and the University of Washington. It is a re-examination of BERT pretraining that measures how hyperparameters and data size affect results, and it ships as downloadable weights and code.

How is it different from BERT?

The main differences are in how pretraining is run rather than in the core idea. RoBERTa redoes its token masking every epoch instead of fixing it once, uses byte-level Byte-Pair Encoding with a 50,000-token vocabulary, and trains far longer on 160GB of text. The authors concluded BERT had been undertrained.

What can you use it for?

It is meant to be fine-tuned on tasks that judge a whole sentence, such as sequence classification, token classification and question answering. You can also run it raw for masked language modeling or to extract text features. For text generation, the model card recommends GPT2 instead.

How do I get it?

The weights and code were released publicly, first in the fairseq repository and later on the Hugging Face hub as FacebookAI/roberta-large. It runs in PyTorch and TensorFlow through the transformers library.

What are its limits?

Training drew on unfiltered internet text, so outputs can reflect social bias, and the model card shows gendered job suggestions as an example. That bias carries over into fine-tuned versions. The model is also case-sensitive and English-only.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on RoBERTa yet, so the facts on this page come from the sources listed above: arxiv.org/abs/1907.11692, huggingface.co/FacebookAI/roberta-large. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.