All models
471 models so far, best known first. 21 more are on the list to be written.
🇺🇸 United StatesGloVe
Unsupervised word embeddings from global word co-occurrence countsResearch · Fully open · 2014
WorldBERT
An encoder-only transformer that reads text in both directions at once.Research · Open weights · 2018
🇺🇸 United StatesRoBERTa
A robustly optimised BERT pretraining study with released models and codeEncoder · Open weights · 2019
🇺🇸 United StatesMegatron-LM
Model-parallel training techniques for transformer language models with billions of parametersResearch · Fully open · 2019
WorldELECTRA
A pre-training method that trains text encoders to spot replaced tokens.Research · Research only · 2020
🇺🇸 United StatesDeBERTa
A BERT-style encoder with disentangled attention and an enhanced mask decoder.Encoder · Open weights · 2020
🇨🇳 ChinaERNIE 1.0
Knowledge-enhanced language model for Chinese NLP tasks.Encoder · Research only · 2019
🇺🇸 United StatesALBERT
A slimmed-down BERT encoder that shares weights across its layersEncoder · Open weights · 2019