News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM

Search: BERT

Concepts

Events

BERT WorldBERT An encoder-only transformer that reads text in both directions at once.Research · Open weights · 2018 ALBERT 🇺🇸 United StatesALBERT A slimmed-down BERT encoder that shares weights across its layersEncoder · Open weights · 2019 ModernBERT 🇺🇸 United States and 1 moreModernBERT A modernised encoder model for retrieval and classification tasks.Encoder · Open weights · 2024 DeBERTa 🇺🇸 United StatesDeBERTa A BERT-style encoder with disentangled attention and an enhanced mask decoder.Encoder · Open weights · 2020 IndicBERT 🇮🇳 IndiaIndicBERT Multilingual encoder model for Indian languagesEncoder · Open weights · 2022 CodeBERT 🇨🇳 ChinaCodeBERT Bimodal pre-trained model for code and textCode · Open weights · 2020 Sentence-BERT 🇩🇪 GermanySentence-BERT Efficient sentence embeddings using Siamese BERT networksEncoder · Open weights · 2019 DistilBERT WorldDistilBERT A smaller, faster version of the BERT language model.Encoder · Open weights · 2019 SciBERT 🇺🇸 United StatesSciBERT A BERT model trained on scientific text.Domain · Open weights · 2019 CamemBERT 🇫🇷 FranceCamemBERT French language model based on RoBERTa architectureEncoder · Open weights · 2019 AfriBERTa WorldAfriBERTa Multilingual encoder model for eleven African languagesEncoder · Open weights · 2021 RoBERTa 🇺🇸 United StatesRoBERTa A robustly optimised BERT pretraining study with released models and codeEncoder · Open weights · 2019 XLM-RoBERTa 🇺🇸 United StatesXLM-RoBERTa A multilingual masked language model covering 100 languages, built on RoBERTa.General · Open weights · 2019 BioBERT WorldBioBERT Biomedical language model based on BERT architectureDomain · Open weights · 2019 XLNet WorldXLNet Permutation-based autoregressive transformer model created by Google AIResearch · Open weights · 2019 wavMamba WorldMamba Efficient sequence modelling architecture using state spacesResearch · Open weights Minerva 🇮🇹 ItalyMinerva An Italian-first large language model trained from scratch at Sapienza University of Rome.General · Open weights · 2024 MuRIL 🇮🇳 IndiaMuRIL A BERT-style encoder pre-trained on 17 Indian languages and their transliterationsEncoder · Open weights · 2021 ELECTRA WorldELECTRA A pre-training method that trains text encoders to spot replaced tokens.Research · Research only · 2020 Codestral Mamba 🇫🇷 FranceCodestral Mamba A 7B open code model built on the Mamba2 architectureCode · Open weights · 2024 Mamba-2 🇺🇸 United StatesMamba-2 A state-space model architecture for efficient language modellingResearch · Fully open · 2024 Megatron-LM 🇺🇸 United StatesMegatron-LM Model-parallel training techniques for transformer language models with billions of parametersResearch · Fully open · 2019

The weekly digest

Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.