Search: BERT
Concepts
Events
WorldBERT
An encoder-only transformer that reads text in both directions at once.Research · Open weights · 2018
🇺🇸 United StatesALBERT
A slimmed-down BERT encoder that shares weights across its layersEncoder · Open weights · 2019
🇺🇸 United States and 1 moreModernBERT
A modernised encoder model for retrieval and classification tasks.Encoder · Open weights · 2024
🇺🇸 United StatesDeBERTa
A BERT-style encoder with disentangled attention and an enhanced mask decoder.Encoder · Open weights · 2020
🇮🇳 IndiaIndicBERT
Multilingual encoder model for Indian languagesEncoder · Open weights · 2022
🇨🇳 ChinaCodeBERT
Bimodal pre-trained model for code and textCode · Open weights · 2020
🇩🇪 GermanySentence-BERT
Efficient sentence embeddings using Siamese BERT networksEncoder · Open weights · 2019
WorldDistilBERT
A smaller, faster version of the BERT language model.Encoder · Open weights · 2019
🇺🇸 United StatesSciBERT
A BERT model trained on scientific text.Domain · Open weights · 2019
🇫🇷 FranceCamemBERT
French language model based on RoBERTa architectureEncoder · Open weights · 2019
WorldAfriBERTa
Multilingual encoder model for eleven African languagesEncoder · Open weights · 2021
🇺🇸 United StatesRoBERTa
A robustly optimised BERT pretraining study with released models and codeEncoder · Open weights · 2019
🇺🇸 United StatesXLM-RoBERTa
A multilingual masked language model covering 100 languages, built on RoBERTa.General · Open weights · 2019
WorldBioBERT
Biomedical language model based on BERT architectureDomain · Open weights · 2019
WorldXLNet
Permutation-based autoregressive transformer model created by Google AIResearch · Open weights · 2019
WorldMamba
Efficient sequence modelling architecture using state spacesResearch · Open weights
🇮🇹 ItalyMinerva
An Italian-first large language model trained from scratch at Sapienza University of Rome.General · Open weights · 2024
🇮🇳 IndiaMuRIL
A BERT-style encoder pre-trained on 17 Indian languages and their transliterationsEncoder · Open weights · 2021
WorldELECTRA
A pre-training method that trains text encoders to spot replaced tokens.Research · Research only · 2020
🇫🇷 FranceCodestral Mamba
A 7B open code model built on the Mamba2 architectureCode · Open weights · 2024
🇺🇸 United StatesMamba-2
A state-space model architecture for efficient language modellingResearch · Fully open · 2024
🇺🇸 United StatesMegatron-LM
Model-parallel training techniques for transformer language models with billions of parametersResearch · Fully open · 2019