Mixture of experts
A design in which each layer holds many expert sub-networks and a router sends each token to just a few of them.
Also: MoE, sparse model
In a mixture-of-experts model, the feed-forward block of each layer is replaced by many parallel experts, from eight to several hundred. A small router network picks the best few for each token. The model can then hold a very large number of parameters while using only a fraction for any one token. Google's 2017 sparsely-gated layer and 2021 Switch Transformer developed the idea, and Mixtral 8x7B, DeepSeek-V3, Qwen3 and Kimi K2 brought it to open models. DeepSeek-V3 has 671 billion parameters in total but uses about 37 billion per token.