News
EveryLLM

Mixture of experts

Mixtral, DeepSeek-V3, Kimi K2: many expert networks, a few used for each token. 112 models on EveryLLM so far.

A mixture of experts (MoE) model replaces each dense feed-forward block with many smaller expert networks and a router that sends each token to only a few of them. The model can hold a very large number of parameters while using only a fraction for any one token, so it learns more without costing proportionally more to run. Mixtral, DeepSeek-V3, Qwen3, Llama 4, gpt-oss and Kimi K2 are all built this way.

Total and active parameters

An MoE has two sizes. DeepSeek-V3 (December 2024) holds 671 billion parameters but activates 37 billion for each token. Mixtral 8x7B (December 2023) has eight experts per layer, uses two for each token, and has 46.7 billion parameters in total with 12.9 billion active. Kimi K2 (July 2025) reaches one trillion in total with 32 billion active. All the experts must still sit in memory, so MoE models need large servers even when they are fast.

A long history

The idea dates to 1991, and Google researchers led by Noam Shazeer revived it for language in 2017 with a sparsely gated layer of thousands of experts. Google's Switch Transformer reached 1.6 trillion parameters in 2021. Open MoE models took off with Mixtral, then Databricks' DBRX (March 2024, 132 billion total, 36 billion active), and became the norm for large Chinese open models. Training them is harder, because routers must spread tokens evenly across experts.

Famous mixture of experts models

Further reading

imgwavvidGemma 馃嚭馃嚫 United StatesGemma Google DeepMind's open family of Gemini-derived language modelsGeneral 路 Open weights 路 2024 imgLlama 馃嚭馃嚫 United StatesLlama Meta AI's family of open-weight language models, from 1B to 2T parametersGeneral 路 Conditional licence 路 2023 imgAlice AI 馃嚪馃嚭 RussiaAlice AI Yandex's family of language, vision and image models behind the Alice assistantGeneral 路 API only 路 2023 DBRX WorldDBRX A 132-billion-parameter mixture-of-experts language model from Databricks.General 路 Conditional licence 路 2024 imgfnMuse 馃嚭馃嚫 United StatesMuse Meta's multimodal LLM family, successor to Llama, with open and closed variantsGeneral 路 API only 路 2026 imgwavvidNemotron 馃嚭馃嚫 United StatesNemotron Nvidia family of language and multimodal models for enterprise and reasoningGeneral 路 Conditional licence 路 2023 fnNemotron 3 Nano 馃嚭馃嚫 United StatesNemotron 3 Nano NVIDIA's 30B hybrid Mamba-Transformer MoE model for reasoning and agentsGeneral 路 Conditional licence 路 2025 Param-1 馃嚠馃嚦 IndiaParam-1 Bilingual and multilingual language models from India.General 路 Conditional licence fnNemotron 3 Super 馃嚭馃嚫 United StatesNemotron 3 Super NVIDIA's 120B hybrid MoE language model with 12B active parameters and 1M-token contextGeneral 路 Conditional licence 路 2026 NLLB-200 馃嚭馃嚫 United StatesNLLB-200 A research translation model covering 200 languages, built for low-resource ones.General 路 Conditional licence 路 2022 PLLuM 馃嚨馃嚤 PolandPLLuM Polish-focused large language model familyGeneral 路 Open weights 路 2024 Solar Open 100B 馃嚢馃嚪 South KoreaSolar Open 100B 102B parameter Mixture-of-Experts model by UpstageGeneral 路 Conditional licence 路 2026 Qwen2 馃嚚馃嚦 ChinaQwen2 Open-weight language model series from AlibabaGeneral 路 Open weights 路 2024

The weekly digest

Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.