News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM

Encoders and embeddings

BERT, RoBERTa, T5 and the models that turn text into meaning for search. 21 models on EveryLLM so far.

Encoders and embedding models turn text into meaning rather than writing new text. An encoder reads a whole passage at once and produces a vector, a list of numbers, for each word or for the passage, placing similar meanings close together. These vectors drive search engines, recommendations, spam filters, document classification and retrieval-augmented generation, in which a chatbot first looks up relevant passages before it answers.

From word vectors to BERT

Word2Vec (Google, 2013) and GloVe (Stanford, 2014) gave each word a single vector learned from its neighbours. ELMo (Allen Institute for AI, 2018) made the vector depend on context, so bank means one thing beside a river and another beside a loan. Google's BERT, published in October 2018, trained a transformer encoder to fill in masked words; BERT-large had 340 million parameters and topped the main language understanding benchmarks of the day.

Embeddings today

BERT spawned a family that includes RoBERTa, DistilBERT and multilingual versions, and Google began using BERT in its search engine in October 2019. Sentence-BERT (2019) adapted it to compare whole sentences quickly. Modern embedding models from OpenAI, Cohere, Google, Nvidia and open projects are judged on benchmarks such as MTEB and often reuse a decoder model's layers. They stay small and cheap, so they can index billions of documents.

Famous encoder models

Still to be written: Word2Vec.

Further reading

BERT WorldBERT An encoder-only transformer that reads text in both directions at once.Research · Open weights · 2018 MuRIL 🇮🇳 IndiaMuRIL A BERT-style encoder pre-trained on 17 Indian languages and their transliterationsEncoder · Open weights · 2021 RoBERTa 🇺🇸 United StatesRoBERTa A robustly optimised BERT pretraining study with released models and codeEncoder · Open weights · 2019 BioBERT WorldBioBERT Biomedical language model based on BERT architectureDomain · Open weights · 2019 DeBERTa 🇺🇸 United StatesDeBERTa A BERT-style encoder with disentangled attention and an enhanced mask decoder.Encoder · Open weights · 2020 IndicBERT 🇮🇳 IndiaIndicBERT Multilingual encoder model for Indian languagesEncoder · Open weights · 2022 Sentence-BERT 🇩🇪 GermanySentence-BERT Efficient sentence embeddings using Siamese BERT networksEncoder · Open weights · 2019 Jina Embeddings v3 🇩🇪 GermanyJina Embeddings v3 Multilingual text embedding model with task-specific adaptersEncoder · Open weights · 2024 imgNomic Embed 🇺🇸 United StatesNomic Embed Open-source long-context text embedding modelEncoder · Open weights · 2024 DistilBERT WorldDistilBERT A smaller, faster version of the BERT language model.Encoder · Open weights · 2019 SciBERT 🇺🇸 United StatesSciBERT A BERT model trained on scientific text.Domain · Open weights · 2019 CamemBERT 🇫🇷 FranceCamemBERT French language model based on RoBERTa architectureEncoder · Open weights · 2019 AfriBERTa WorldAfriBERTa Multilingual encoder model for eleven African languagesEncoder · Open weights · 2021 ALBERT 🇺🇸 United StatesALBERT A slimmed-down BERT encoder that shares weights across its layersEncoder · Open weights · 2019 ModernBERT 🇺🇸 United States and 1 moreModernBERT A modernised encoder model for retrieval and classification tasks.Encoder · Open weights · 2024

The weekly digest

Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.