News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM

Research milestones

GPT-1 and GPT-2, ELMo, XLNet, Megatron, Chinchilla: models that changed the field. 67 models on EveryLLM so far.

Research milestones are models that changed how the field works, whether or not anyone still uses them. Some introduced a method, such as pre-training followed by fine-tuning; some showed what scale could do; others corrected earlier thinking, such as the 2022 finding that most large models had been trained on too little data. Many were never products, and several are now retired, but their papers are still widely read.

The pre-training years

ELMo (February 2018) and OpenAI's GPT-1 (June 2018, 117 million parameters) showed that a model pre-trained on plain text could be fine-tuned cheaply for many tasks. BERT followed in October 2018. GPT-2 (February 2019, 1.5 billion parameters) wrote fluent paragraphs, and OpenAI at first withheld the full model over misuse concerns, releasing it in stages up to November 2019. XLNet (June 2019) and T5 (October 2019) refined the recipe.

Scale and its rules

GPT-3 (2020, 175 billion parameters) showed few-shot learning: doing a new task from examples in the prompt. In 2021 the Beijing Academy of Artificial Intelligence announced Wu Dao 2.0 at 1.75 trillion parameters. DeepMind's Chinchilla (March 2022) showed that a 70-billion-parameter model trained on 1.4 trillion tokens beat much larger models, resetting how labs balance model size against the amount of training data.

Open recipes

In March 2023 Stanford's Alpaca showed that a 7-billion-parameter Llama model, tuned on 52,000 instruction examples, could behave much like a commercial chatbot for under US$600. EleutherAI's GPT-J (2021) and the BLOOM collaboration (2022) proved that groups outside the big labs could train and release capable models, a thread that led to the open-weights ecosystem of today.

Famous research models

Further reading

ELECTRA WorldELECTRA A pre-training method that trains text encoders to spot replaced tokens.Research · Research only · 2020 RETRO WorldRETRO A retrieval-enhanced transformer that conditions on chunks from a huge document database.Research · Research only · 2021 fnToolformer 🇺🇸 United StatesToolformer A language model that learns on its own when to call external tools through APIs.Research · Research only · 2023 imgPaLM-E WorldPaLM-E An embodied multimodal language model that grounds words in robot sensor inputMultimodal · Research only · 2023 Galactica 🇺🇸 United StatesGalactica A language model trained on scientific text for research tasksDomain · Conditional licence · 2022 imgMM1 WorldMM1 A family of multimodal language models up to 30B parameters, dense and MoE.Multimodal · Research only · 2024 AlexaTM 20B 🇺🇸 United StatesAlexaTM 20B A 20-billion-parameter multilingual sequence-to-sequence model from Amazon's Alexa team.General · Research only · 2022 Orca 2 🇺🇸 United StatesOrca 2 A small LLaMA-2-based model trained on synthetic data to reason.Small · Conditional licence · 2023 Phi-2 🇺🇸 United StatesPhi-2 A 2.7 billion parameter Microsoft research model for QA, chat and codeResearch · Open weights InstructGPT 🇺🇸 United StatesInstructGPT GPT-3 models fine-tuned with human feedback to follow instructionsGeneral · API only · 2022 fnWebGPT 🇺🇸 United StatesWebGPT A GPT-3 model fine-tuned to answer questions while browsing the webGeneral · Research only · 2021 RecurrentGemma 🇺🇸 United StatesRecurrentGemma Open language models using Google's Griffin recurrent-plus-attention architectureGeneral · Open weights · 2024 Med-PaLM 🇺🇸 United StatesMed-PaLM A medical-domain language model aligned from Flan-PaLM using instruction prompt tuning.Domain · Research only · 2022 Switch Transformer WorldSwitch Transformer A sparse mixture-of-experts language model scaling to a trillion parameters.Research · Research only · 2021 GLaM 🇺🇸 United StatesGLaM A trillion-parameter mixture-of-experts language model from Google ResearchResearch · Research only · 2021 Minerva 🇺🇸 United StatesMinerva A language model trained further on technical content to solve maths and science problemsResearch · Research only · 2022 ALBERT 🇺🇸 United StatesALBERT A slimmed-down BERT encoder that shares weights across its layersEncoder · Open weights · 2019 imgvidFlamingo 🇬🇧 United KingdomFlamingo A visual language model family that learns new image and video tasks from a few examples.Multimodal · Research only · 2022 Gopher 🇬🇧 United KingdomGopher A 280 billion parameter Transformer language model from DeepMind, studied for how scale changes ability.Research · Research only · 2021

The weekly digest

Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.