RETRO
A retrieval-enhanced transformer that conditions on chunks from a huge document database.
RETRO (Retrieval-Enhanced Transformer) is an auto-regressive language model that conditions its predictions on document chunks pulled from a 2 trillion token database. A team including Sebastian Borgeaud, Arthur Mensch and Jack W. Rae described it in a paper first posted in December 2021, showing performance comparable to GPT-3 and Jurassic-1 on the Pile with 25 times fewer parameters.
- Retrieves document chunks from a 2 trillion token database
- Matches GPT-3 and Jurassic-1 on the Pile with 25x fewer parameters
- Uses a frozen BERT retriever plus chunked cross-attention
- Can be retrofitted onto pre-trained transformers
- First posted to arXiv on 8 December 2021
What it is
RETRO, short for Retrieval-Enhanced Transformer, is an auto-regressive language model that decides what to write next partly by looking up relevant text. Rather than relying only on what it absorbed during training, it retrieves document chunks from a database of 2 trillion tokens, matching them against the tokens that come just before. The paper describing it was first posted on 8 December 2021. The authors report that RETRO reached results on the Pile benchmark comparable to GPT-3 and Jurassic-1 while using 25 times fewer parameters, and that after fine-tuning it carried those gains into knowledge-heavy tasks such as question answering.
How it came about
The work was written up by 28 authors, among them Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Jack W. Rae, Laurent Sifre and Karen Simonyan. Version 1 of the paper appeared on arXiv on 8 December 2021, with a revision in January 2022 and a further version on 7 February 2022 that corrected numbers in one of its tables. The team notes that RETRO is usually trained from the ground up, but that existing transformers can also be retrofitted with retrieval quickly and still perform well.
Inside the model
RETRO works by conditioning each prediction on chunks of documents retrieved from a very large corpus, chosen for local similarity with the preceding tokens. Three parts do this: a frozen BERT retriever that finds candidate text, a differentiable encoder that turns those chunks into usable representations, and a chunked cross-attention mechanism that lets the decoder consult them while generating. The setup lets the model draw on roughly an order of magnitude more text than it consumes during ordinary training. Because retrieval supplies the extra knowledge, the model itself can stay far smaller than models that must memorise everything in their weights.
What it can do
On the Pile, a standard language modelling benchmark, the authors report performance comparable to GPT-3 and Jurassic-1 despite using 25 times fewer parameters. After fine-tuning, that performance carried over to knowledge-intensive downstream work such as question answering. The paper presents retrieval as a way to add explicit memory to language models at very large scale.
How to use it
The work is published as an arXiv paper with a PDF available. The abstract describes the method and results; the listing points to code and media search tools but the page itself does not state a licence or a released set of weights.
Why it matters
The authors argue that conditioning a language model on retrieved text opens a route to better models through explicit memory at unprecedented scale. They also show that the approach can be added to transformers that already exist, rather than requiring a full retrain, which widens the set of models that could use it.
Questions people ask about RETRO
What does RETRO stand for?
RETRO is short for Retrieval-Enhanced Transformer. It is an auto-regressive language model that conditions its predictions on document chunks retrieved from a large corpus, chosen by local similarity with the tokens that come before.
How is it different from a normal language model?
A normal model stores knowledge in its parameters. RETRO also looks things up at run time, pulling matching chunks from a 2 trillion token database and attending to them through a chunked cross-attention mechanism, so it can draw on far more text than it saw in training.
How does it compare to GPT-3?
The authors report that RETRO obtains performance comparable to GPT-3 and Jurassic-1 on the Pile while using 25 times fewer parameters. After fine-tuning, that performance transfers to knowledge-intensive tasks such as question answering.
Can it be added to models that already exist?
Yes. The team says RETRO is typically trained from scratch, but that pre-trained transformers can be rapidly retrofitted with retrieval and still achieve good performance.
Good reading and sources
There is no English Wikipedia article on RETRO yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2112.04426. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.