CamemBERT
French language model based on RoBERTa architecture
CamemBERT is a pretrained language model designed specifically for French text processing. Developed by researchers including Louis Martin and Benoît Sagot, it uses the RoBERTa architecture to improve performance on tasks like parsing and entity recognition compared to multilingual alternatives.
- Based on RoBERTa architecture for French text
- Trained on OSCAR and CCNet web corpora
- Available in 110M and 335M parameter sizes
- Improves state of the art in French NLP tasks
- Small 4GB datasets matched larger 130GB results
What it is
CamemBERT is a language model built for processing French text. It adapts the RoBERTa architecture, which is a robustly optimized version of BERT, to focus solely on French rather than mixing multiple languages. The project offers several variants, ranging from 110 million to 335 million parameters. These models are trained on large corpora such as OSCAR and CCNet, aiming to provide better results for French-specific natural language processing tasks than general multilingual models.
How it came about
The model was introduced in a paper submitted to arXiv in November 2019 and later presented at the Association for Computational Linguistics meeting in July 2020. A team of eight researchers, including Louis Martin and Benoît Sagot, developed the system. They investigated whether training a monolingual transformer on French data would outperform existing multilingual options. Their work demonstrated that web-crawled data could yield strong results even in smaller quantities compared to larger datasets.
Inside the model
The architecture follows the encoder-only design of RoBERTa. It uses a sub-word tokenizer based on SentencePiece to break French text into manageable units. Training data sources include the OSCAR corpus, containing 138 gigabytes of text, and CCNet, with 135 gigabytes. The researchers also tested smaller subsets, such as four gigabytes from Wikipedia or subsampled versions of OSCAR and CCNet. The model generates contextual embeddings, allowing downstream applications to capture meaning based on surrounding words.
What it can do
CamemBERT improves or matches the state of the art in four key areas: part-of-speech tagging, dependency parsing, named entity recognition, and natural language inference. The developers found that web-crawled data performed better than Wikipedia-only data. Surprisingly, a small dataset of four gigabytes produced results comparable to those from much larger collections exceeding 130 gigabytes. This suggests efficient learning from high-quality web text for French linguistic structures.
How to use it
The model weights are available openly through the Hugging Face platform. Users can access six different versions, varying by parameter count and training data source. Integration is supported via the Transformers library, allowing developers to load the model and its tokenizer easily. The code examples show how to use pipelines for tasks like filling masked tokens or extracting embedding features for further analysis.
Reception and debate
The work was accepted as a long paper for the 2020 Annual Meeting of the Association for Computational Linguistics. It addressed a gap in natural language processing tools for languages other than English. By focusing on French, the team provided a resource that practical users had lacked, as most prior models were either English-centric or diluted across many languages.
Why it matters
CamemBERT demonstrated the viability of monolingual models for non-English languages. It showed that high-quality web crawls could serve as effective training data, challenging the assumption that massive scale is always necessary for good performance. This approach has influenced subsequent efforts to build dedicated language models for other specific tongues, encouraging the use of targeted data over broad multilingual mixes.
Questions people ask about CamemBERT
What is CamemBERT used for?
It is used for French natural language processing tasks such as part-of-speech tagging, dependency parsing, named entity recognition, and natural language inference. It provides contextual embeddings for French text.
How large is the CamemBERT model?
It comes in two main sizes. The base version has 110 million parameters, while the large version has 335 million parameters. Both are available with different training data sources.
Is CamemBERT multilingual?
No, it is a monolingual model trained specifically on French data. This focus allows it to perform better on French tasks than multilingual models that dilute their capacity across many languages.
Good reading and sources
There is no English Wikipedia article on CamemBERT yet, so the facts on this page come from the sources listed above: huggingface.co/almanach/camembert-base, arxiv.org/abs/1911.03894. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.