ModernBERT
A modernised encoder model for retrieval and classification tasks.
ModernBERT is a bidirectional encoder-only transformer released in December 2024 by Answer.AI and LightOn. It processes up to 8,192 tokens of English text and code, offering improved speed and accuracy for search and classification compared to earlier BERT models.
- Trained on 2 trillion tokens of English text and code.
- Supports a native context length of up to 8,192 tokens.
- Released under the permissive Apache 2.0 licence.
- Achieves state-of-the-art results in code retrieval tasks.
- Uses Flash Attention and rotary positional embeddings.
What it is
ModernBERT is a bidirectional encoder-only transformer model designed for tasks such as retrieval, classification and semantic search. Released in December 2024, it was developed through a collaboration between Answer.AI, LightOn and other contributors. The model comes in two sizes: base with 149 million parameters and large with 395 million parameters. It supports a native context window of up to 8,192 tokens, allowing it to process long documents efficiently. Training data consists of two trillion tokens drawn from English text and computer code.
How it came about
The model represents an update to the original BERT architecture, which has seen few major improvements since its initial release. The development team aimed to apply modern optimisation techniques to encoder-only models to achieve better performance and efficiency. The project was announced via a pre-print paper on arXiv in December 2024, titled Smarter, Better, Faster, Longer. The authors include researchers from Answer.AI and LightOn, who sought to create a Pareto improvement over older encoders.
Inside the model
ModernBERT uses an encoder-only design with Pre-Norm Transformer layers and GeGLU activation functions. It incorporates Rotary Positional Embeddings to support long contexts and employs local-global alternating attention for efficiency. The training process utilised unpadding and Flash Attention to speed up inference. The model was trained on eight H100 graphics processing units using the StableAdamW optimizer. Sequence length was initially set to 1,024 tokens before being extended to 8,192 tokens during training.
What it can do
The model performs strongly on natural language understanding benchmarks such as GLUE, where the large version scores second only to Deberta-v3-large. It achieves state-of-the-art results in code retrieval tasks on CodeSearchNet and StackQA due to its mixed training data. In general retrieval tests like BEIR, it outperforms similarly sized models in both single-vector and multi-vector settings. The developers note that performance may be lower for languages other than English, as the training data is primarily English and code.
How to use it
Model weights, architectures and training code are released under the Apache 2.0 licence, making them free for commercial and research use. Users can access the models via the Hugging Face Hub and integrate them using the transformers library version 4.48.0 or later. The developers recommend using Flash Attention 2 on compatible hardware to maximise efficiency. Unlike some earlier BERT variants, ModernBERT does not require token type IDs for downstream tasks.
Safety and testing
The developers state that the model may produce representations reflecting biases present in its training data. They advise users to verify critical or sensitive outputs before relying on them for decision-making. No specific red-teaming results or safety evaluations beyond standard benchmark testing are detailed in the provided materials.
Reception and debate
The release was accompanied by a technical paper claiming significant improvements in speed and memory efficiency over previous encoder models. The authors describe the model as the most speed and memory efficient encoder designed for inference on common graphics processing units. Evaluation tables show it surpassing older models like RoBERTa and NomicBERT in several retrieval and code tasks.
Why it matters
ModernBERT demonstrates that encoder-only models can still benefit significantly from modern architectural choices and extensive pretraining. Its ability to handle long contexts and code data efficiently provides a strong backbone for hybrid semantic search applications. The open licence encourages further adoption and fine-tuning for specific downstream tasks such as question answering and document retrieval.
Questions people ask about ModernBERT
What is the maximum context length for ModernBERT?
ModernBERT supports a native context length of up to 8,192 tokens. This allows it to process long documents efficiently for tasks such as retrieval and classification within large corpora.
Is ModernBERT available for commercial use?
Yes, the model weights, architectures and training code are released under the Apache 2.0 licence. This permissive licence allows for both commercial and research usage without restrictive conditions.
What languages does ModernBERT support?
The model was trained primarily on English text and computer code. Consequently, its performance may be lower for other languages, and it is best suited for English-language tasks.
How does ModernBERT differ from original BERT?
It uses modern architectural improvements such as rotary positional embeddings, local-global alternating attention and GeGLU activations. It also supports much longer context windows and was trained on a larger dataset including code.
Good reading and sources
There is no English Wikipedia article on ModernBERT yet, so the facts on this page come from the sources listed above: huggingface.co/answerdotai/ModernBERT-large, arxiv.org/abs/2412.13663. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.