News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
Hunyuan-Large

Hunyuan-Large

Open-source mixture of experts model with 389 billion parameters

Tencent released Hunyuan-Large in November 2024 as the largest open-source Transformer-based mixture of experts model. It features 389 billion total parameters with 52 billion active per token, supporting long contexts and strong multilingual performance.

Share on WhatsApp

  • 389 billion total parameters with 52 billion active
  • Supports context windows up to 256,000 tokens
  • Outperforms Llama3.1-70B on most benchmarks
  • Uses Grouped Query Attention for efficiency
  • Trained with large-scale synthetic data

What it is

Hunyuan-Large is a large language model developed by Tencent. Released in November 2024, it uses a mixture of experts architecture to balance size and efficiency. The model contains 389 billion parameters in total but activates only 52 billion during inference. This design allows it to handle complex tasks while managing computational resources. It supports text sequences up to 256,000 tokens in its pre-trained version. The model aims to advance open-source artificial intelligence research and application.

How it came about

Tencent introduced Hunyuan-Large in November 2024 via a technical report and arXiv paper. The team sought to address resource consumption challenges in large models by exploring mixture of experts designs. They released the model weights and code to encourage community innovation. The publication detailed training practices and benchmark results to guide future development. This release marked a significant step in open-source large model availability.

Inside the model

The model employs a Transformer-based mixture of experts structure. It uses Grouped Query Attention and Cross-Layer Attention to compress key-value caches, reducing memory use. Training involved expert-specific learning rates to ensure each sub-model contributes effectively. The developers utilized large-scale synthetic data to enhance representation learning. These techniques help the model generalize better to unseen data and handle long inputs efficiently.

What it can do

Hunyuan-Large performs strongly across language understanding, reasoning, and mathematics benchmarks. It outperforms Llama3.1-70B and rivals Llama3.1-405B on many tasks. Scores include 88.4 on MMLU and 92.8 on GSM8K for the pre-trained version. The instruct model achieves 89.9 on MMLU and 77.4 on MATH. It shows particular strength in Chinese tasks like C-Eval and CMMLU. Coding performance reaches 90.0 on HumanEval for the instruct variant.

How to use it

The model weights and code are open-source. Developers can access them through Hugging Face and Gitee. Deployment tools include vLLM, with TRT-LLM support planned. Tencent provides a quick start guide and API access via HunyuanAPI. This open approach allows researchers and engineers to deploy and optimize the model locally or in cloud environments.

Safety and testing

The team conducted extensive experiments to validate safety across various languages and tasks. Benchmark evaluations included checks for practical effectiveness and robustness. The technical report highlights these safety measures as part of the model development process. No specific incidents or red-teaming details are provided in the available materials.

Reception and debate

The model is presented as a leading open-source option in its class. Benchmarks show it competitive with larger dense models from other labs. The release aims to inspire researchers and advance AI technology collectively. Community engagement is encouraged through open-source platforms and shared technical details.

Why it matters

By releasing such a large mixture of experts model openly, Tencent provides a reference for efficient scaling. The technical details on synthetic data and cache compression offer insights for future model optimization. This contribution supports the broader ecosystem of open-weight language models.

Questions people ask about Hunyuan-Large

How many parameters does Hunyuan-Large have?

The model has 389 billion total parameters. However, it uses a mixture of experts architecture, so only 52 billion parameters are active during any given inference step. This makes it more efficient than dense models of similar total size.

What is the maximum context length?

The pre-trained model supports text sequences up to 256,000 tokens. The instruct version supports up to 128,000 tokens. This allows the model to process very long documents or codebases in a single pass.

Is Hunyuan-Large open source?

Yes, Tencent released the model weights and code as open-source. You can download them from Hugging Face or Gitee. The company also provides deployment tools like vLLM to help users run the model efficiently.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on Hunyuan-Large yet, so the facts on this page come from the sources listed above: huggingface.co/tencent/Tencent-Hunyuan-Large, arxiv.org/abs/2411.02265. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.