News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
DeepSeekMoE

DeepSeekMoE

Mixture-of-experts language model architecture by DeepSeek AI

DeepSeekMoE is a mixture-of-experts architecture designed to improve expert specialisation and reduce computational costs. Released in early 2024 by DeepSeek AI, it scales from 2 billion to 16 billion parameters while maintaining performance comparable to larger dense models.

Share on WhatsApp

  • Uses mixture-of-experts architecture for efficiency
  • 16B model matches LLaMA2 7B with 40% compute
  • Introduces shared experts to reduce redundancy
  • Weights available under commercial-friendly license
  • Scales effectively from 2B to 145B parameters

What it is

DeepSeekMoE is a language model architecture developed by DeepSeek AI that utilises a mixture-of-experts approach. This design aims to manage computational costs while scaling up model parameters. The system was introduced in January 2024 through a research paper. It focuses on ensuring that each expert within the model acquires focused and non-overlapping knowledge. The architecture has been implemented in models ranging from 2 billion to 16 billion parameters, with preliminary scaling to 145 billion parameters.

How it came about

The architecture was proposed in a paper published on arXiv in January 2024. The work was conducted by a team of researchers including Damai Dai and Chengqi Deng. The development started with a modest scale of 2 billion parameters to demonstrate viability against existing architectures like GShard. Subsequent efforts scaled the model to 16 billion parameters, showing competitive performance with LLaMA2 7B. Preliminary work also explored scaling to 145 billion parameters.

Inside the model

The core innovation involves two strategies for expert specialisation. First, it finely segments experts into a larger number and activates a flexible combination of them. Second, it isolates a subset of experts as shared ones to capture common knowledge and reduce redundancy in routed experts. This structure allows the model to achieve comparable performance to dense counterparts with significantly fewer computations. For instance, the 16 billion parameter version uses about 40 per cent of the computation required by LLaMA2 7B.

What it can do

The 2 billion parameter variant achieves performance comparable to GShard 2.9 billion, despite having fewer expert parameters and computation. It nearly approaches the performance of its dense counterpart with the same total parameters. The 16 billion parameter model matches LLaMA2 7B performance with reduced computational load. Preliminary results for the 145 billion parameter model show advantages over GShard and performance comparable to DeepSeek 67 billion, using only 28.5 per cent or potentially 18.2 per cent of computations.

How to use it

The code repository is available under the MIT License. The use of the model weights is subject to a specific Model License, which supports commercial use. Weights for the 16 billion parameter base model are hosted on Hugging Face. Users can interact with the model via provided Python scripts using libraries such as Transformers.

Questions people ask about DeepSeekMoE

What is the main advantage of DeepSeekMoE?

It reduces computational costs while scaling parameters by ensuring expert specialisation. This allows smaller active parameter counts to match the performance of larger dense models.

Can I use DeepSeekMoE for commercial projects?

Yes, the model license supports commercial use. The code repository is licensed under the MIT License, but users must adhere to the specific Model License for the weights.

How does it differ from standard Mixture-of-Experts models?

It finely segments experts and isolates shared experts to capture common knowledge. This design mitigates redundancy and improves the specialisation of individual routed experts.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on DeepSeekMoE yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2401.06066, huggingface.co/deepseek-ai/deepseek-moe-16b-base. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.