News
ResearchOpenAI pens AI safety agreement with four firmsReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierOpenAI’s New Hacking Model Found Dozens Of Holes In Apple TechFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIResearchOpenAI pens AI safety agreement with four firmsReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierOpenAI’s New Hacking Model Found Dozens Of Holes In Apple TechFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAI
EveryLLM

Encoders and embeddings

BERT, RoBERTa, T5 and the models that turn text into meaning for search. 20 models on EveryLLM so far.

Encoders and embedding models turn text into meaning rather than writing new text. An encoder reads a whole passage at once and produces a vector, a list of numbers, for each word or for the passage, placing similar meanings close together. These vectors drive search engines, recommendations, spam filters, document classification and retrieval-augmented generation, in which a chatbot first looks up relevant passages before it answers.

From word vectors to BERT

Word2Vec (Google, 2013) and GloVe (Stanford, 2014) gave each word a single vector learned from its neighbours. ELMo (Allen Institute for AI, 2018) made the vector depend on context, so bank means one thing beside a river and another beside a loan. Google's BERT, published in October 2018, trained a transformer encoder to fill in masked words; BERT-large had 340 million parameters and topped the main language understanding benchmarks of the day.

Embeddings today

BERT spawned a family that includes RoBERTa, DistilBERT and multilingual versions, and Google began using BERT in its search engine in October 2019. Sentence-BERT (2019) adapted it to compare whole sentences quickly. Modern embedding models from OpenAI, Cohere, Google, Nvidia and open projects are judged on benchmarks such as MTEB and often reuse a decoder model's layers. They stay small and cheap, so they can index billions of documents.

Famous encoder models

Still to be written: Word2Vec.

Further reading

The weekly digest

Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.