News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
ELMo

ELMo

A 2018 bidirectional LSTM that produced context-sensitive word embeddings.

ELMo (embeddings from language model) is a word embedding method that turns a sequence of words into a sequence of vectors with context-sensitive meaning. Researchers at the Allen Institute for Artificial Intelligence and the University of Washington first released it in February 2018. It helped pioneer pretraining followed by fine-tuning and pointed the field toward transformer-based language models.

Share on WhatsApp

  • First released in February 2018 by AI2 and the University of Washington
  • Bidirectional LSTM reading character-level input into word-level embeddings
  • Trained on about 30 million sentences, roughly 1 billion words
  • Five components joined into a 512-dimensional vector per token
  • Reported improvements on six benchmark NLP tasks in the original paper

What it is

ELMo stands for embeddings from language model. It is a method for turning a run of words into a matching run of vectors, so that each word gets a representation shaped by the words around it. It was built by researchers at the Allen Institute for Artificial Intelligence and the University of Washington and first released in February 2018. The system is a bidirectional LSTM, a kind of recurrent network that reads a sequence in both directions. It takes character-level input and emits word-level embeddings. Training used roughly 30 million sentences, about 1 billion words in total. Its main use is deep contextualised word representation, which helps tasks such as working out what a pronoun refers to and which sense of a word is meant.

How it came about

ELMo came out of work at the Allen Institute for Artificial Intelligence together with the University of Washington, and appeared in February 2018. It sits in a line of methods for representing words. Earlier options counted word frequencies, or learned one fixed vector per word from co-occurrence patterns, as GloVe and Word2Vec did. ELMo instead gave the same spelling different vectors depending on context, and applied bidirectional LSTMs to this problem at large scale. After the Transformer architecture was published in 2017, the design was reworked into a Transformer encoder, which produced BERT.

Inside the model

ELMo is a stack of bidirectional LSTMs sitting on a token embedding layer. Text is first mapped to vectors by that embedding layer. Two branches then run over it in parallel. The forward branch is a 2-layer LSTM with 4096 units and 512-dimension projections, with a residual link from the first layer to the second. The backward branch matches it but reads the sequence from end to start. Outputs from all five parts, the embedding layer plus two forward and two backward LSTM layers, are joined and multiplied by a projection matrix to give a 512-dimensional vector per token. Pretraining used 1 billion words: the forward side predicts the next token, the backward side the previous one. Afterwards the weights are frozen except the projection matrix, which is fine-tuned for each task.

What it can do

Because each layer sees text from one side only, ELMo builds up context in stages. For the sentence "She went to the bank to withdraw money", the first forward LSTM reads "bank" against "She went to the", suggesting a destination, while the first backward LSTM reads it against "to withdraw money", pointing to a financial institution. The second forward layer can then combine both readings. This kind of deep contextualised representation helps with coreference resolution and polysemy resolution. The original paper reported gains over the previous best results on six benchmark NLP tasks.

How to use it

ELMo is described as a research method rather than a shipped product. Its weights were pretrained on a text corpus and then adapted per task by fine-tuning the projection matrix. No app, API or licence terms are described here.

Why it matters

ELMo is treated as an important early step in a historical progression of language modelling. It helped establish the pattern of self-supervised generative pretraining followed by fine-tuning: a large model learns to reproduce a big corpus, then extra task-specific weights are added and tuned on labelled data. Unlike bag-of-words embeddings such as Word2Vec and GloVe, its vectors change with context. It was an instrumental step toward transformer-based language modelling, and its pretrain-then-fine-tune workflow carried over into BERT, which swapped the LSTM stack for a Transformer encoder and trained more in parallel.

Questions people ask about ELMo

What does ELMo actually do?

It converts a sequence of words into a matching sequence of vectors. Unlike earlier word embeddings, the vector for a word changes depending on the surrounding text, so the same spelling can get different representations in different sentences.

Who created it and when?

Researchers at the Allen Institute for Artificial Intelligence and the University of Washington built it, and it was first released in February 2018.

How is it trained?

It is pretrained on about 1 billion words, around 30 million sentences. The forward branch repeatedly predicts the next token and the backward branch the previous one. Afterwards most weights are frozen and only the projection matrix is fine-tuned per task.

How does it differ from Word2Vec or GloVe?

Those methods assign each word one fixed vector based on co-occurrence patterns. ELMo produces context-sensitive embeddings, so a word like "bank" gets different vectors depending on whether the sentence is about money or a river.

How does it relate to BERT?

After the Transformer appeared in 2017, ELMo's bidirectional LSTM design was replaced by a Transformer encoder, giving rise to BERT. BERT kept the pretrain-then-fine-tune workflow but trains with more parallelism.

Good reading and sources

Where this page comes from

The facts on this page come from the Wikipedia article "ELMo", written by its contributors and used under the CC BY-SA 4.0 licence, and from its Wikidata record (CC0). EveryLLM rewrote the text in its own words, with AI assistance, and compares every page with the article to make sure no passage is copied. It may be reused under the same CC BY-SA 4.0 licence with a link back.

It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

The weekly digest

Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.