News
ResearchOpenAI pens AI safety agreement with four firmsReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierOpenAI鈥檚 New Hacking Model Found Dozens Of Holes In Apple TechFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIResearchOpenAI pens AI safety agreement with four firmsReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierOpenAI鈥檚 New Hacking Model Found Dozens Of Holes In Apple TechFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAI
EveryLLM

Dense decoder transformers

The GPT design: one stack of transformer layers predicting the next token. 308 models on EveryLLM so far.

The dense decoder transformer is the design behind GPT and most chatbots. It is a single stack of identical transformer layers that reads the tokens so far and predicts the next one, with a mask that stops each position from looking ahead. Dense means every parameter is used for every token, unlike in a mixture of experts. GPT-3, PaLM, Llama 3, Chinchilla, Gemma, Phi and OLMo are dense decoders.

Inside each layer

Each layer has two parts. Self-attention lets every token weigh every earlier token and gather what it needs; a feed-forward network then transforms each token's vector on its own. The transformer was introduced by Google researchers in the June 2017 paper Attention Is All You Need, for translation, and OpenAI's GPT-1 (2018) kept only its decoder half. Llama 3.1 405B, released in July 2024, stacks 126 such layers.

Why it won

A decoder trains on any text without labels, because the next token is always the answer to predict, and it scales smoothly: scaling laws published by OpenAI in 2020 and DeepMind in 2022 showed the error falling predictably with more parameters, data and compute. Its weak point is cost. Attention grows with the square of the context length, and every parameter must be read for every token, which is why mixtures of experts became popular.

Famous dense decoder models

Still to be written: OLMo (language model).

Further reading

GPT-Neo WorldGPT-Neo Open autoregressive language model by EleutherAIGeneral 路 Open weights ALIA-40b 馃嚜馃嚫 SpainALIA-40b Multilingual base model for 35 European languagesGeneral 路 Open weights OLMo 2 馃嚭馃嚫 United StatesOLMo 2 Open language model family with full training transparencyGeneral 路 Open weights 路 2024 OLMo 3 馃嚭馃嚫 United StatesOLMo 3 Open language models for scientific research by Allen Institute for AIGeneral 路 Open weights 路 2025 T眉lu 3 馃嚭馃嚫 United StatesT眉lu 3 Open post-trained model family built on Llama 3.1General 路 Open weights 路 2024 Qwen-72B 馃嚚馃嚦 ChinaQwen-72B A 72-billion parameter open-weight language model from Alibaba Cloud.General 路 Open weights 路 2023 Qwen1.5 馃嚚馃嚦 ChinaQwen1.5 Open-weight language model series with nine sizes and strong multilingual support.General 路 Open weights 路 2024 Pharia-1-LLM-7B 馃嚛馃嚜 GermanyPharia-1-LLM-7B Multilingual 7B parameter decoder model by Aleph Alpha ResearchGeneral 路 Open weights CroissantLLM 馃嚝馃嚪 FranceCroissantLLM A 1.3 billion parameter bilingual French-English language model.General 路 Open weights 路 2024 Lucie-7B 馃嚝馃嚪 FranceLucie-7B Open multilingual language model focused on French and EnglishGeneral 路 Open weights 路 2025 PhoGPT 馃嚮馃嚦 VietnamPhoGPT Open-source generative model for Vietnamese language tasks.General 路 Open weights 路 2023 Bielik 馃嚨馃嚤 PolandBielik Polish-optimised 11 billion parameter language modelGeneral 路 Open weights 路 2025 BLOOMZ WorldBLOOMZ Multilingual instruction-following model based on BLOOMGeneral 路 Open weights 路 2022 fnQwQ-32B 馃嚚馃嚦 ChinaQwQ-32B A 32-billion parameter reasoning model from the Qwen series.Reasoning 路 Open weights 路 2025 PLaMo-100B 馃嚡馃嚨 JapanPLaMo-100B A 100-billion parameter language model built for Japanese.General 路 Open weights 路 2024 fnMotif-2 WorldMotif-2 A 12.7 billion parameter reasoning model by Motif Technologies.Reasoning 路 Open weights EvoLLM-JP 馃嚡馃嚨 JapanEvoLLM-JP Experimental Japanese language model created by merging open-source models.General 路 Open weights 路 2024 SeaLLMs 馃嚫馃嚞 SingaporeSeaLLMs Open language models for Southeast Asian languagesGeneral 路 Open weights 路 2023 MaLLaM 馃嚥馃嚲 MalaysiaMaLLaM Large language models trained on Malaysian context and Malay language data.General 路 Open weights 路 2024 YaLM 100B 馃嚪馃嚭 RussiaYaLM 100B Open-weight Russian and English language model with 100 billion parametersGeneral 路 Open weights EXAONE 3.0 馃嚢馃嚪 South KoreaEXAONE 3.0 Open bilingual language model from LG AI ResearchGeneral 路 Open weights 路 2024 Kanana 馃嚢馃嚪 South KoreaKanana Bilingual language models for Korean and English by Kakao.General 路 Open weights 路 2025 EuroLLM WorldEuroLLM Open multilingual models for European languagesGeneral 路 Open weights 路 2024 Falcon-40B 馃嚘馃嚜 United Arab EmiratesFalcon-40B A 40-billion parameter open language model from TII.General 路 Open weights 路 2023 LLM-jp-3 馃嚡馃嚨 JapanLLM-jp-3 Open Japanese language model family by NIIGeneral 路 Open weights 路 2024 Stockmark-100b 馃嚡馃嚨 JapanStockmark-100b A 100 billion parameter Japanese and English language model.General 路 Open weights MiniCPM 馃嚚馃嚦 ChinaMiniCPM Efficient small language models for end-side devicesGeneral 路 Open weights 路 2024 Krutrim-2 馃嚠馃嚦 IndiaKrutrim-2 A 12-billion parameter multilingual language model focused on Indian languages and context.General 路 Open weights 路 2024 fnSeed-OSS-36B 馃嚚馃嚦 ChinaSeed-OSS-36B Open-source 36B parameter language model with controllable reasoning.General 路 Open weights 路 2025 Baichuan 2 馃嚚馃嚦 ChinaBaichuan 2 Open multilingual language models with 7B and 13B parameters.General 路 Open weights 路 2023 imgNVLM 1.0 馃嚭馃嚫 United StatesNVLM 1.0 Open multimodal language models from NVIDIA for vision and language tasksMultimodal 路 Open weights 路 2024 Fugaku-LLM 馃嚡馃嚨 JapanFugaku-LLM Japanese large language model trained on Fugaku supercomputerGeneral 路 Open weights 路 2024 K2-Think WorldK2-Think A 32 billion parameter open reasoning modelReasoning 路 Open weights 路 2025 Qwen2.5-72B 馃嚚馃嚦 ChinaQwen2.5-72B A 72-billion parameter open-weight language model from Alibaba Cloud.General 路 Open weights 路 2024 CodeGen 馃嚭馃嚫 United StatesCodeGen Open autoregressive language model for code generationCode 路 Open weights 路 2022 EXAONE Deep 馃嚢馃嚪 South KoreaEXAONE Deep Reasoning-focused language models from LG AI ResearchReasoning 路 Open weights 路 2025 DeepSeek Coder 馃嚚馃嚦 ChinaDeepSeek Coder Open-source code generation models from DeepSeekCode 路 Open weights 路 2024 fnMistral Small 3 馃嚝馃嚪 FranceMistral Small 3 A 24B open-weight model tuned for low latency and local deploymentGeneral 路 Open weights 路 2025 Qwen2 馃嚚馃嚦 ChinaQwen2 Open-weight language model series from AlibabaGeneral 路 Open weights 路 2024 Qwen2.5-Coder 馃嚚馃嚦 ChinaQwen2.5-Coder Open code-specialised language model series by Alibaba CloudCode 路 Open weights 路 2024 Sailor 馃嚫馃嚞 SingaporeSailor Open multilingual language models for South-East AsiaGeneral 路 Open weights 路 2024 imgPixtral 12B 馃嚝馃嚪 FrancePixtral 12B A 12B multimodal model that reads images and text without losing text skillsMultimodal 路 Open weights 路 2024 fnMistral NeMo 馃嚝馃嚪 FranceMistral NeMo A 12B multilingual open-weights model built with NVIDIA, with a 128k context window.General 路 Open weights 路 2024 Aya Expanse WorldAya Expanse Open-weight multilingual language models in 8B and 32B sizes from Cohere For AIGeneral 路 Open weights 路 2024 imgSalamandra 馃嚜馃嚫 SpainSalamandra Open multilingual language models for European languagesGeneral 路 Open weights 路 2025 imgvidfnInternVL3 馃嚚馃嚦 ChinaInternVL3 Open-source multimodal large language model seriesMultimodal 路 Open weights 路 2025 Granite 3.0 馃嚭馃嚫 United StatesGranite 3.0 IBM's third-generation open enterprise language models, from 2B to 8B parametersGeneral 路 Open weights 路 2024 Falcon-7B 馃嚘馃嚜 United Arab EmiratesFalcon-7B Open 7-billion parameter decoder model by TIIGeneral 路 Open weights 路 2023

The weekly digest

Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.