News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM

Research milestones

GPT-1 and GPT-2, ELMo, XLNet, Megatron, Chinchilla: models that changed the field. 67 models on EveryLLM so far.

Research milestones are models that changed how the field works, whether or not anyone still uses them. Some introduced a method, such as pre-training followed by fine-tuning; some showed what scale could do; others corrected earlier thinking, such as the 2022 finding that most large models had been trained on too little data. Many were never products, and several are now retired, but their papers are still widely read.

The pre-training years

ELMo (February 2018) and OpenAI's GPT-1 (June 2018, 117 million parameters) showed that a model pre-trained on plain text could be fine-tuned cheaply for many tasks. BERT followed in October 2018. GPT-2 (February 2019, 1.5 billion parameters) wrote fluent paragraphs, and OpenAI at first withheld the full model over misuse concerns, releasing it in stages up to November 2019. XLNet (June 2019) and T5 (October 2019) refined the recipe.

Scale and its rules

GPT-3 (2020, 175 billion parameters) showed few-shot learning: doing a new task from examples in the prompt. In 2021 the Beijing Academy of Artificial Intelligence announced Wu Dao 2.0 at 1.75 trillion parameters. DeepMind's Chinchilla (March 2022) showed that a 70-billion-parameter model trained on 1.4 trillion tokens beat much larger models, resetting how labs balance model size against the amount of training data.

Open recipes

In March 2023 Stanford's Alpaca showed that a 7-billion-parameter Llama model, tuned on 52,000 instruction examples, could behave much like a commercial chatbot for under US$600. EleutherAI's GPT-J (2021) and the BLOOM collaboration (2022) proved that groups outside the big labs could train and release capable models, a thread that led to the open-weights ecosystem of today.

Famous research models

Further reading

1.58-bit large language model World1.58-bit large language model A model whose weights hold only three values: -1, 0 and +1.Research · Open weights · 2024 Chinchilla 🇬🇧 United KingdomChinchilla A 70-billion-parameter DeepMind model built to test how models should scale.Research · Research only · 2022 Stanford Alpaca 🇺🇸 United StatesStanford Alpaca A cheap instruction-tuned LLaMA 7B built at Stanford in 2023Research · Research only · 2023 imgwavvidVideoPoet 🇺🇸 United StatesVideoPoet A Google Research language model that turns text and images into video and audioMultimodal · Research only · 2023 fnSparrow 🇬🇧 United KingdomSparrow DeepMind's rule-following English chatbot that cites web evidence for its answersAssistant · Research only · 2022 Gopher 🇬🇧 United KingdomGopher A 280 billion parameter Transformer language model from DeepMind, studied for how scale changes ability.Research · Research only · 2021 DialoGPT 🇺🇸 United StatesDialoGPT A GPT-style model trained on Reddit threads to write chat repliesGeneral · Open weights · 2019 OPT 🇺🇸 United StatesOPT Meta AI's family of open decoder-only language models from 125M to 175B parametersGeneral · Conditional licence · 2022 AlphaCode 🇬🇧 United KingdomAlphaCode A code generation system that solves competitive programming problems.Code · Research only · 2022 RETRO WorldRETRO A retrieval-enhanced transformer that conditions on chunks from a huge document database.Research · Research only · 2021 imgMM1 WorldMM1 A family of multimodal language models up to 30B parameters, dense and MoE.Multimodal · Research only · 2024 fnToolformer 🇺🇸 United StatesToolformer A language model that learns on its own when to call external tools through APIs.Research · Research only · 2023 Orca 2 🇺🇸 United StatesOrca 2 A small LLaMA-2-based model trained on synthetic data to reason.Small · Conditional licence · 2023 fnWebGPT 🇺🇸 United StatesWebGPT A GPT-3 model fine-tuned to answer questions while browsing the webGeneral · Research only · 2021 Med-PaLM 🇺🇸 United StatesMed-PaLM A medical-domain language model aligned from Flan-PaLM using instruction prompt tuning.Domain · Research only · 2022 Minerva 🇺🇸 United StatesMinerva A language model trained further on technical content to solve maths and science problemsResearch · Research only · 2022

The weekly digest

Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.