DeepSeek-V3
Open-weight mixture of experts language model with 671 billion parameters.
DeepSeek-V3 is a large language model released by DeepSeek-AI in late 2024. It uses a mixture of experts architecture with 671 billion total parameters but activates only 37 billion per token. The model was trained on 14.8 trillion tokens and offers open weights for research and commercial use.
- 671 billion total parameters with 37 billion activated per token
- Trained on 14.8 trillion tokens using 2.788 million H800 GPU hours
- Supports a context window of 128,000 tokens
- Uses FP8 mixed precision for efficient training
- Distills reasoning capabilities from the DeepSeek-R1 model
What it is
DeepSeek-V3 is a general-purpose language model developed by DeepSeek-AI. It features a mixture of experts design with 671 billion total parameters, though only 37 billion are active for any given token. Released in December 2024, it supports a context window of 128,000 tokens. The model handles text in multiple languages, including English and Chinese, and aims to match the performance of leading closed-source systems while remaining open for public access.
How it came about
The model builds upon the architecture validated in DeepSeek-V2. DeepSeek-AI announced the release in late 2024 via a technical report. The development team focused on improving training efficiency and stability compared to previous iterations. The project involved a large group of researchers and engineers who worked to scale up the model size without incurring prohibitive computational costs.
Inside the model
The system uses Multi-head Latent Attention and a custom Mixture of Experts structure. It employs an auxiliary-loss-free strategy to balance load across experts without hurting performance. Training used a multi-token prediction objective and FP8 mixed precision. The team pre-trained the model on 14.8 trillion tokens using 2.788 million H800 GPU hours. Post-training involved supervised fine-tuning and reinforcement learning, including knowledge distillation from the DeepSeek-R1 reasoning model.
What it can do
Benchmarks show strong results in code generation, mathematics, and multilingual tasks. It scored 87.5 on BBH and 65.2 on HumanEval. In Chinese evaluations, it reached 90.1 on C-Eval. The chat version achieved 59.1 on GPQA-Diamond and 39.2 on AIME 2024. It performs well on long-context tests up to its 128K limit. The model demonstrates competitive capabilities against larger dense models and proprietary systems.
How to use it
Weights are available for download on Hugging Face. The release includes both base and chat versions. The total file size is approximately 685 billion parameters when including the multi-token prediction module. Developers can run the model locally with support from community tools and hardware vendors. The licence permits broad usage, encouraging integration into various applications.
Safety and testing
The training process was reported as stable, with no irrecoverable loss spikes or rollbacks. The developers incorporated verification and reflection patterns from the R1 series to improve reasoning reliability. They maintained control over output style and length during the distillation process. Specific red-teaming results or external safety audits are not detailed in the provided technical report.
Reception and debate
The model received attention for its cost-effective training and high performance relative to its active parameter count. Evaluations suggest it outperforms other open-source models in many categories. It is viewed as a significant step for open-weight models in competing with closed systems. The technical report highlights its efficiency as a key advantage for the broader community.
Why it matters
DeepSeek-V3 demonstrates that large-scale mixture of experts models can be trained efficiently using FP8 precision. Its success validates the architectural choices made in the V2 series. The open release provides a strong baseline for further research in load balancing and multi-token prediction. It influences how developers approach scaling laws and training stability for massive models.
Questions people ask about DeepSeek-V3
How many parameters does DeepSeek-V3 have?
The model has 671 billion total parameters. However, it uses a mixture of experts architecture, so only 37 billion parameters are activated for each token during inference.
What is the context length of DeepSeek-V3?
DeepSeek-V3 supports a context window of 128,000 tokens. This allows it to process and generate text based on very long documents or conversations.
Is DeepSeek-V3 open source?
Yes, the model weights are open and available for download on platforms like Hugging Face. It is not fully open source in terms of training data, but the weights are accessible for use and modification.
How much compute was needed to train DeepSeek-V3?
The full training process required 2.788 million H800 GPU hours. This includes 2.664 million hours for pre-training and 0.1 million hours for subsequent stages.
Good reading and sources
There is no English Wikipedia article on DeepSeek-V3 yet, so the facts on this page come from the sources listed above: huggingface.co/deepseek-ai/DeepSeek-V3, arxiv.org/abs/2412.19437. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.