DeepSeekMoE
Mixture-of-experts language model architecture by DeepSeek AI
DeepSeekMoE is a mixture-of-experts architecture designed to improve expert specialisation and reduce computational costs. Released in early 2024 by DeepSeek AI, it scales from 2 billion to 16 billion parameters while maintaining performance comparable to larger dense models.
- Uses mixture-of-experts architecture for efficiency
- 16B model matches LLaMA2 7B with 40% compute
- Introduces shared experts to reduce redundancy
- Weights available under commercial-friendly license
- Scales effectively from 2B to 145B parameters
What it is
DeepSeekMoE is a language model architecture developed by DeepSeek AI that utilises a mixture-of-experts approach. This design aims to manage computational costs while scaling up model parameters. The system was introduced in January 2024 through a research paper. It focuses on ensuring that each expert within the model acquires focused and non-overlapping knowledge. The architecture has been implemented in models ranging from 2 billion to 16 billion parameters, with preliminary scaling to 145 billion parameters.
How it came about
The architecture was proposed in a paper published on arXiv in January 2024. The work was conducted by a team of researchers including Damai Dai and Chengqi Deng. The development started with a modest scale of 2 billion parameters to demonstrate viability against existing architectures like GShard. Subsequent efforts scaled the model to 16 billion parameters, showing competitive performance with LLaMA2 7B. Preliminary work also explored scaling to 145 billion parameters.
Inside the model
The core innovation involves two strategies for expert specialisation. First, it finely segments experts into a larger number and activates a flexible combination of them. Second, it isolates a subset of experts as shared ones to capture common knowledge and reduce redundancy in routed experts. This structure allows the model to achieve comparable performance to dense counterparts with significantly fewer computations. For instance, the 16 billion parameter version uses about 40 per cent of the computation required by LLaMA2 7B.
What it can do
The 2 billion parameter variant achieves performance comparable to GShard 2.9 billion, despite having fewer expert parameters and computation. It nearly approaches the performance of its dense counterpart with the same total parameters. The 16 billion parameter model matches LLaMA2 7B performance with reduced computational load. Preliminary results for the 145 billion parameter model show advantages over GShard and performance comparable to DeepSeek 67 billion, using only 28.5 per cent or potentially 18.2 per cent of computations.
How to use it
The code repository is available under the MIT License. The use of the model weights is subject to a specific Model License, which supports commercial use. Weights for the 16 billion parameter base model are hosted on Hugging Face. Users can interact with the model via provided Python scripts using libraries such as Transformers.
Questions people ask about DeepSeekMoE
What is the main advantage of DeepSeekMoE?
It reduces computational costs while scaling parameters by ensuring expert specialisation. This allows smaller active parameter counts to match the performance of larger dense models.
Can I use DeepSeekMoE for commercial projects?
Yes, the model license supports commercial use. The code repository is licensed under the MIT License, but users must adhere to the specific Model License for the weights.
How does it differ from standard Mixture-of-Experts models?
It finely segments experts and isolates shared experts to capture common knowledge. This design mitigates redundancy and improves the specialisation of individual routed experts.
Good reading and sources
There is no English Wikipedia article on DeepSeekMoE yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2401.06066, huggingface.co/deepseek-ai/deepseek-moe-16b-base. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.