News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
DeepSeek-V3

DeepSeek-V3

Open-weight mixture of experts language model with 671 billion parameters.

DeepSeek-V3 is a large language model released by DeepSeek-AI in late 2024. It uses a mixture of experts architecture with 671 billion total parameters but activates only 37 billion per token. The model was trained on 14.8 trillion tokens and offers open weights for research and commercial use.

Share on WhatsApp

  • 671 billion total parameters with 37 billion activated per token
  • Trained on 14.8 trillion tokens using 2.788 million H800 GPU hours
  • Supports a context window of 128,000 tokens
  • Uses FP8 mixed precision for efficient training
  • Distills reasoning capabilities from the DeepSeek-R1 model

What it is

DeepSeek-V3 is a general-purpose language model developed by DeepSeek-AI. It features a mixture of experts design with 671 billion total parameters, though only 37 billion are active for any given token. Released in December 2024, it supports a context window of 128,000 tokens. The model handles text in multiple languages, including English and Chinese, and aims to match the performance of leading closed-source systems while remaining open for public access.

How it came about

The model builds upon the architecture validated in DeepSeek-V2. DeepSeek-AI announced the release in late 2024 via a technical report. The development team focused on improving training efficiency and stability compared to previous iterations. The project involved a large group of researchers and engineers who worked to scale up the model size without incurring prohibitive computational costs.

Inside the model

The system uses Multi-head Latent Attention and a custom Mixture of Experts structure. It employs an auxiliary-loss-free strategy to balance load across experts without hurting performance. Training used a multi-token prediction objective and FP8 mixed precision. The team pre-trained the model on 14.8 trillion tokens using 2.788 million H800 GPU hours. Post-training involved supervised fine-tuning and reinforcement learning, including knowledge distillation from the DeepSeek-R1 reasoning model.

What it can do

Benchmarks show strong results in code generation, mathematics, and multilingual tasks. It scored 87.5 on BBH and 65.2 on HumanEval. In Chinese evaluations, it reached 90.1 on C-Eval. The chat version achieved 59.1 on GPQA-Diamond and 39.2 on AIME 2024. It performs well on long-context tests up to its 128K limit. The model demonstrates competitive capabilities against larger dense models and proprietary systems.

How to use it

Weights are available for download on Hugging Face. The release includes both base and chat versions. The total file size is approximately 685 billion parameters when including the multi-token prediction module. Developers can run the model locally with support from community tools and hardware vendors. The licence permits broad usage, encouraging integration into various applications.

Safety and testing

The training process was reported as stable, with no irrecoverable loss spikes or rollbacks. The developers incorporated verification and reflection patterns from the R1 series to improve reasoning reliability. They maintained control over output style and length during the distillation process. Specific red-teaming results or external safety audits are not detailed in the provided technical report.

Reception and debate

The model received attention for its cost-effective training and high performance relative to its active parameter count. Evaluations suggest it outperforms other open-source models in many categories. It is viewed as a significant step for open-weight models in competing with closed systems. The technical report highlights its efficiency as a key advantage for the broader community.

Why it matters

DeepSeek-V3 demonstrates that large-scale mixture of experts models can be trained efficiently using FP8 precision. Its success validates the architectural choices made in the V2 series. The open release provides a strong baseline for further research in load balancing and multi-token prediction. It influences how developers approach scaling laws and training stability for massive models.

Questions people ask about DeepSeek-V3

How many parameters does DeepSeek-V3 have?

The model has 671 billion total parameters. However, it uses a mixture of experts architecture, so only 37 billion parameters are activated for each token during inference.

What is the context length of DeepSeek-V3?

DeepSeek-V3 supports a context window of 128,000 tokens. This allows it to process and generate text based on very long documents or conversations.

Is DeepSeek-V3 open source?

Yes, the model weights are open and available for download on platforms like Hugging Face. It is not fully open source in terms of training data, but the weights are accessible for use and modification.

How much compute was needed to train DeepSeek-V3?

The full training process required 2.788 million H800 GPU hours. This includes 2.664 million hours for pre-training and 0.1 million hours for subsequent stages.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on DeepSeek-V3 yet, so the facts on this page come from the sources listed above: huggingface.co/deepseek-ai/DeepSeek-V3, arxiv.org/abs/2412.19437. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.