Hunyuan-Large
Open-source mixture of experts model with 389 billion parameters
Tencent released Hunyuan-Large in November 2024 as the largest open-source Transformer-based mixture of experts model. It features 389 billion total parameters with 52 billion active per token, supporting long contexts and strong multilingual performance.
- 389 billion total parameters with 52 billion active
- Supports context windows up to 256,000 tokens
- Outperforms Llama3.1-70B on most benchmarks
- Uses Grouped Query Attention for efficiency
- Trained with large-scale synthetic data
What it is
Hunyuan-Large is a large language model developed by Tencent. Released in November 2024, it uses a mixture of experts architecture to balance size and efficiency. The model contains 389 billion parameters in total but activates only 52 billion during inference. This design allows it to handle complex tasks while managing computational resources. It supports text sequences up to 256,000 tokens in its pre-trained version. The model aims to advance open-source artificial intelligence research and application.
How it came about
Tencent introduced Hunyuan-Large in November 2024 via a technical report and arXiv paper. The team sought to address resource consumption challenges in large models by exploring mixture of experts designs. They released the model weights and code to encourage community innovation. The publication detailed training practices and benchmark results to guide future development. This release marked a significant step in open-source large model availability.
Inside the model
The model employs a Transformer-based mixture of experts structure. It uses Grouped Query Attention and Cross-Layer Attention to compress key-value caches, reducing memory use. Training involved expert-specific learning rates to ensure each sub-model contributes effectively. The developers utilized large-scale synthetic data to enhance representation learning. These techniques help the model generalize better to unseen data and handle long inputs efficiently.
What it can do
Hunyuan-Large performs strongly across language understanding, reasoning, and mathematics benchmarks. It outperforms Llama3.1-70B and rivals Llama3.1-405B on many tasks. Scores include 88.4 on MMLU and 92.8 on GSM8K for the pre-trained version. The instruct model achieves 89.9 on MMLU and 77.4 on MATH. It shows particular strength in Chinese tasks like C-Eval and CMMLU. Coding performance reaches 90.0 on HumanEval for the instruct variant.
How to use it
The model weights and code are open-source. Developers can access them through Hugging Face and Gitee. Deployment tools include vLLM, with TRT-LLM support planned. Tencent provides a quick start guide and API access via HunyuanAPI. This open approach allows researchers and engineers to deploy and optimize the model locally or in cloud environments.
Safety and testing
The team conducted extensive experiments to validate safety across various languages and tasks. Benchmark evaluations included checks for practical effectiveness and robustness. The technical report highlights these safety measures as part of the model development process. No specific incidents or red-teaming details are provided in the available materials.
Reception and debate
The model is presented as a leading open-source option in its class. Benchmarks show it competitive with larger dense models from other labs. The release aims to inspire researchers and advance AI technology collectively. Community engagement is encouraged through open-source platforms and shared technical details.
Why it matters
By releasing such a large mixture of experts model openly, Tencent provides a reference for efficient scaling. The technical details on synthetic data and cache compression offer insights for future model optimization. This contribution supports the broader ecosystem of open-weight language models.
Questions people ask about Hunyuan-Large
How many parameters does Hunyuan-Large have?
The model has 389 billion total parameters. However, it uses a mixture of experts architecture, so only 52 billion parameters are active during any given inference step. This makes it more efficient than dense models of similar total size.
What is the maximum context length?
The pre-trained model supports text sequences up to 256,000 tokens. The instruct version supports up to 128,000 tokens. This allows the model to process very long documents or codebases in a single pass.
Is Hunyuan-Large open source?
Yes, Tencent released the model weights and code as open-source. You can download them from Hugging Face or Gitee. The company also provides deployment tools like vLLM to help users run the model efficiently.
Good reading and sources
There is no English Wikipedia article on Hunyuan-Large yet, so the facts on this page come from the sources listed above: huggingface.co/tencent/Tencent-Hunyuan-Large, arxiv.org/abs/2411.02265. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.