XLM-RoBERTa
A multilingual masked language model covering 100 languages, built on RoBERTa.
XLM-RoBERTa is a multilingual masked language model released in November 2019 by a team at Facebook AI. Trained on filtered CommonCrawl text in 100 languages, it showed that one model could match strong single-language systems while transferring well to low-resource languages.
- Trained on 2.5TB of filtered CommonCrawl text in 100 languages
- XNLI average accuracy improved by 14.6% over multilingual BERT
- Swahili XNLI accuracy rose 15.7% over earlier XLM models
- Uses masked language modelling with 15% of words hidden
- First released through the fairseq repository in November 2019
What it is
XLM-RoBERTa, often shortened to XLM-R, is a multilingual language model that learns representations of text in 100 languages at once. A team at Facebook AI introduced it in November 2019. It is a masked language model: parts of an input sentence are hidden and the system predicts them, which teaches it to read both left and right context. The large version is the one published on Hugging Face. Rather than generating text, it is meant to be adapted to tasks such as classifying sentences, labelling tokens and answering questions, where its output features feed a smaller classifier.
How it came about
The model appeared in the paper Unsupervised Cross-lingual Representation Learning at Scale, posted to arXiv on 5 November 2019 and revised in April 2020. Ten researchers are listed as authors, led by Alexis Conneau. Code and weights were first put out through the fairseq repository. The Hugging Face model card notes that the original team did not write one, so Hugging Face staff produced it instead.
Inside the model
XLM-RoBERTa adapts RoBERTa, a Transformer trained without human labels, to many languages. The corpus is more than two terabytes of CommonCrawl web text, filtered and covering 100 languages. The paper examines two tensions: gains from sharing across languages against the loss of capacity per language, and how high-resource and low-resource languages behave as data grows.
What it can do
Reported results show sizeable gains over multilingual BERT: average accuracy on XNLI rose by 14.6%, average F1 on MLQA by 13%, and F1 on named entity recognition by 2.4%. Improvements were largest for languages with little training data, with XNLI accuracy up 15.7% for Swahili and 11.4% for Urdu compared with earlier XLM models. The authors also claim competitiveness with strong single-language models on GLUE and XNLI. The model is not designed for text generation.
How to use it
Weights for the large version are hosted on Hugging Face under the FacebookAI account, alongside code in the fairseq repository. The team said code, data and models would be released publicly. The model can be run directly for filling in masked words or loaded to extract features, and many fine-tuned variants exist on the model hub.
Why it matters
The work is presented as the first demonstration that a single multilingual model need not give up accuracy in any one language. Its analysis of capacity dilution and of the balance between high- and low-resource languages informed later multilingual training. The released weights became a common starting point for cross-lingual transfer work.
Questions people ask about XLM-RoBERTa
What is XLM-RoBERTa for?
It is a base model meant to be fine-tuned on tasks that judge whole sentences or tokens, such as classification, token labelling and question answering. It can also fill in masked words directly, but it is not built for generating text.
How is it different from RoBERTa?
RoBERTa is trained on one language, while XLM-RoBERTa extends the same masked-language-modelling approach to 100 languages using over two terabytes of filtered CommonCrawl web text.
How large is the training data?
The model card gives 2.5TB of filtered CommonCrawl text covering 100 languages, while the paper describes it as more than two terabytes. Parameter counts are not given in these pages.
How well does it work on languages with little data?
The paper reports the biggest gains there: XNLI accuracy improved by 15.7% for Swahili and 11.4% for Urdu compared with earlier XLM models.
Where can I get it?
The large version is downloadable from Hugging Face, and the original code lives in the fairseq repository. The authors said code, data and models would be made public.
Good reading and sources
There is no English Wikipedia article on XLM-RoBERTa yet, so the facts on this page come from the sources listed above: arxiv.org/abs/1911.02116, huggingface.co/FacebookAI/xlm-roberta-large. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.