The 2010s
22 models released in the 2010s, year by year.
2018
WorldBERT
An encoder-only transformer that reads text in both directions at once.Research · Open weights · 2018
🇺🇸 United StatesELMo
A 2018 bidirectional LSTM that produced context-sensitive word embeddings.Research · Research only · 2018
WorldGPT-1
OpenAI's first GPT model, a 117M-parameter decoder-only transformer from 2018.Research · Open weights · 2018
WorldULMFiT
Transfer learning method for NLP text classification tasksResearch · Open weights · 2018
2019
🇺🇸 United StatesALBERT
A slimmed-down BERT encoder that shares weights across its layersEncoder · Open weights · 2019
🇺🇸 United StatesBART
A denoising encoder-decoder transformer for generation and comprehension tasksGeneral · Conditional licence · 2019
WorldBioBERT
Biomedical language model based on BERT architectureDomain · Open weights · 2019
🇫🇷 FranceCamemBERT
French language model based on RoBERTa architectureEncoder · Open weights · 2019
🇺🇸 United StatesDialoGPT
A GPT-style model trained on Reddit threads to write chat repliesGeneral · Open weights · 2019
WorldDistilBERT
A smaller, faster version of the BERT language model.Encoder · Open weights · 2019
🇨🇳 ChinaERNIE 1.0
Knowledge-enhanced language model for Chinese NLP tasks.Encoder · Research only · 2019
🇺🇸 United StatesGPT-2
OpenAI's 2019 text model that predicts the next word in a sequenceGeneral · Open weights · 2019
🇺🇸 United StatesMegatron-LM
Model-parallel training techniques for transformer language models with billions of parametersResearch · Fully open · 2019
🇺🇸 United StatesRoBERTa
A robustly optimised BERT pretraining study with released models and codeEncoder · Open weights · 2019
🇺🇸 United StatesSciBERT
A BERT model trained on scientific text.Domain · Open weights · 2019
🇩🇪 GermanySentence-BERT
Efficient sentence embeddings using Siamese BERT networksEncoder · Open weights · 2019
🇺🇸 United StatesT5
Encoder-decoder language model family from Google AI using a text-to-text framework.General · Fully open · 2019
🇺🇸 United StatesXLM-RoBERTa
A multilingual masked language model covering 100 languages, built on RoBERTa.General · Open weights · 2019
WorldXLNet
Permutation-based autoregressive transformer model created by Google AIResearch · Open weights · 2019