ELMo
A 2018 bidirectional LSTM that produced context-sensitive word embeddings.
ELMo (embeddings from language model) is a word embedding method that turns a sequence of words into a sequence of vectors with context-sensitive meaning. Researchers at the Allen Institute for Artificial Intelligence and the University of Washington first released it in February 2018. It helped pioneer pretraining followed by fine-tuning and pointed the field toward transformer-based language models.
- First released in February 2018 by AI2 and the University of Washington
- Bidirectional LSTM reading character-level input into word-level embeddings
- Trained on about 30 million sentences, roughly 1 billion words
- Five components joined into a 512-dimensional vector per token
- Reported improvements on six benchmark NLP tasks in the original paper
What it is
ELMo stands for embeddings from language model. It is a method for turning a run of words into a matching run of vectors, so that each word gets a representation shaped by the words around it. It was built by researchers at the Allen Institute for Artificial Intelligence and the University of Washington and first released in February 2018. The system is a bidirectional LSTM, a kind of recurrent network that reads a sequence in both directions. It takes character-level input and emits word-level embeddings. Training used roughly 30 million sentences, about 1 billion words in total. Its main use is deep contextualised word representation, which helps tasks such as working out what a pronoun refers to and which sense of a word is meant.
How it came about
ELMo came out of work at the Allen Institute for Artificial Intelligence together with the University of Washington, and appeared in February 2018. It sits in a line of methods for representing words. Earlier options counted word frequencies, or learned one fixed vector per word from co-occurrence patterns, as GloVe and Word2Vec did. ELMo instead gave the same spelling different vectors depending on context, and applied bidirectional LSTMs to this problem at large scale. After the Transformer architecture was published in 2017, the design was reworked into a Transformer encoder, which produced BERT.
Inside the model
ELMo is a stack of bidirectional LSTMs sitting on a token embedding layer. Text is first mapped to vectors by that embedding layer. Two branches then run over it in parallel. The forward branch is a 2-layer LSTM with 4096 units and 512-dimension projections, with a residual link from the first layer to the second. The backward branch matches it but reads the sequence from end to start. Outputs from all five parts, the embedding layer plus two forward and two backward LSTM layers, are joined and multiplied by a projection matrix to give a 512-dimensional vector per token. Pretraining used 1 billion words: the forward side predicts the next token, the backward side the previous one. Afterwards the weights are frozen except the projection matrix, which is fine-tuned for each task.
What it can do
Because each layer sees text from one side only, ELMo builds up context in stages. For the sentence "She went to the bank to withdraw money", the first forward LSTM reads "bank" against "She went to the", suggesting a destination, while the first backward LSTM reads it against "to withdraw money", pointing to a financial institution. The second forward layer can then combine both readings. This kind of deep contextualised representation helps with coreference resolution and polysemy resolution. The original paper reported gains over the previous best results on six benchmark NLP tasks.
How to use it
ELMo is described as a research method rather than a shipped product. Its weights were pretrained on a text corpus and then adapted per task by fine-tuning the projection matrix. No app, API or licence terms are described here.
Why it matters
ELMo is treated as an important early step in a historical progression of language modelling. It helped establish the pattern of self-supervised generative pretraining followed by fine-tuning: a large model learns to reproduce a big corpus, then extra task-specific weights are added and tuned on labelled data. Unlike bag-of-words embeddings such as Word2Vec and GloVe, its vectors change with context. It was an instrumental step toward transformer-based language modelling, and its pretrain-then-fine-tune workflow carried over into BERT, which swapped the LSTM stack for a Transformer encoder and trained more in parallel.
Questions people ask about ELMo
What does ELMo actually do?
It converts a sequence of words into a matching sequence of vectors. Unlike earlier word embeddings, the vector for a word changes depending on the surrounding text, so the same spelling can get different representations in different sentences.
Who created it and when?
Researchers at the Allen Institute for Artificial Intelligence and the University of Washington built it, and it was first released in February 2018.
How is it trained?
It is pretrained on about 1 billion words, around 30 million sentences. The forward branch repeatedly predicts the next token and the backward branch the previous one. Afterwards most weights are frozen and only the projection matrix is fine-tuned per task.
How does it differ from Word2Vec or GloVe?
Those methods assign each word one fixed vector based on co-occurrence patterns. ELMo produces context-sensitive embeddings, so a word like "bank" gets different vectors depending on whether the sentence is about money or a river.
How does it relate to BERT?
After the Transformer appeared in 2017, ELMo's bidirectional LSTM design was replaced by a Transformer encoder, giving rise to BERT. BERT kept the pretrain-then-fine-tune workflow but trains with more parallelism.
Good reading and sources
The facts on this page come from the Wikipedia article "ELMo", written by its contributors and used under the CC BY-SA 4.0 licence, and from its Wikidata record (CC0). EveryLLM rewrote the text in its own words, with AI assistance, and compares every page with the article to make sure no passage is copied. It may be reused under the same CC BY-SA 4.0 licence with a link back.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.