News
EveryLLM

Encoders and embeddings

BERT, RoBERTa, T5 and the models that turn text into meaning for search. 20 models on EveryLLM so far.

Encoders and embedding models turn text into meaning rather than writing new text. An encoder reads a whole passage at once and produces a vector, a list of numbers, for each word or for the passage, placing similar meanings close together. These vectors drive search engines, recommendations, spam filters, document classification and retrieval-augmented generation, in which a chatbot first looks up relevant passages before it answers.

From word vectors to BERT

Word2Vec (Google, 2013) and GloVe (Stanford, 2014) gave each word a single vector learned from its neighbours. ELMo (Allen Institute for AI, 2018) made the vector depend on context, so bank means one thing beside a river and another beside a loan. Google's BERT, published in October 2018, trained a transformer encoder to fill in masked words; BERT-large had 340 million parameters and topped the main language understanding benchmarks of the day.

Embeddings today

BERT spawned a family that includes RoBERTa, DistilBERT and multilingual versions, and Google began using BERT in its search engine in October 2019. Sentence-BERT (2019) adapted it to compare whole sentences quickly. Modern embedding models from OpenAI, Cohere, Google, Nvidia and open projects are judged on benchmarks such as MTEB and often reuse a decoder model's layers. They stay small and cheap, so they can index billions of documents.

Famous encoder models

Still to be written: Word2Vec.

Further reading

The weekly digest

Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.