Research milestones
GPT-1 and GPT-2, ELMo, XLNet, Megatron, Chinchilla: models that changed the field. 68 models on EveryLLM so far.
Research milestones are models that changed how the field works, whether or not anyone still uses them. Some introduced a method, such as pre-training followed by fine-tuning; some showed what scale could do; others corrected earlier thinking, such as the 2022 finding that most large models had been trained on too little data. Many were never products, and several are now retired, but their papers are still widely read.
The pre-training years
ELMo (February 2018) and OpenAI's GPT-1 (June 2018, 117 million parameters) showed that a model pre-trained on plain text could be fine-tuned cheaply for many tasks. BERT followed in October 2018. GPT-2 (February 2019, 1.5 billion parameters) wrote fluent paragraphs, and OpenAI at first withheld the full model over misuse concerns, releasing it in stages up to November 2019. XLNet (June 2019) and T5 (October 2019) refined the recipe.
Scale and its rules
GPT-3 (2020, 175 billion parameters) showed few-shot learning: doing a new task from examples in the prompt. In 2021 the Beijing Academy of Artificial Intelligence announced Wu Dao 2.0 at 1.75 trillion parameters. DeepMind's Chinchilla (March 2022) showed that a 70-billion-parameter model trained on 1.4 trillion tokens beat much larger models, resetting how labs balance model size against the amount of training data.
Open recipes
In March 2023 Stanford's Alpaca showed that a 7-billion-parameter Llama model, tuned on 52,000 instruction examples, could behave much like a commercial chatbot for under US$600. EleutherAI's GPT-J (2021) and the BLOOM collaboration (2022) proved that groups outside the big labs could train and release capable models, a thread that led to the open-weights ecosystem of today.
Famous research models
Further reading
Nothing matches those filters yet. Take one off, or check back: new pages are written every day.