Encoders and embeddings
BERT, RoBERTa, T5 and the models that turn text into meaning for search. 20 models on EveryLLM so far.
Encoders and embedding models turn text into meaning rather than writing new text. An encoder reads a whole passage at once and produces a vector, a list of numbers, for each word or for the passage, placing similar meanings close together. These vectors drive search engines, recommendations, spam filters, document classification and retrieval-augmented generation, in which a chatbot first looks up relevant passages before it answers.
From word vectors to BERT
Word2Vec (Google, 2013) and GloVe (Stanford, 2014) gave each word a single vector learned from its neighbours. ELMo (Allen Institute for AI, 2018) made the vector depend on context, so bank means one thing beside a river and another beside a loan. Google's BERT, published in October 2018, trained a transformer encoder to fill in masked words; BERT-large had 340 million parameters and topped the main language understanding benchmarks of the day.
Embeddings today
BERT spawned a family that includes RoBERTa, DistilBERT and multilingual versions, and Google began using BERT in its search engine in October 2019. Sentence-BERT (2019) adapted it to compare whole sentences quickly. Modern embedding models from OpenAI, Cohere, Google, Nvidia and open projects are judged on benchmarks such as MTEB and often reuse a decoder model's layers. They stay small and cheap, so they can index billions of documents.
Famous encoder models
Still to be written: Word2Vec.
Further reading
Nothing matches those filters yet. Take one off, or check back: new pages are written every day.