Pangu Ultra
A 135-billion parameter dense language model trained on Ascend hardware.
Pangu Ultra is a large language model with 135 billion parameters, developed by Huawei researchers. Released in April 2025, it was trained on Ascend Neural Processing Units to demonstrate the capability of dense architectures at scale. The model aims to compete with larger sparse models through optimized training techniques and extensive post-training for reasoning tasks.
- Contains 135 billion parameters in a dense structure.
- Trained on 13.2 trillion high-quality tokens.
- Uses 8,192 Ascend NPUs for efficient training.
- Employs depth-scaled sandwich normalization to stabilize training.
- Available exclusively to commercial customers.
What it is
Pangu Ultra is a dense large language model containing 135 billion parameters. It was created by a team at Huawei and described in a paper submitted in April 2025. The model uses standard Transformer modules rather than sparse mixture-of-experts structures. Its primary purpose is to push the limits of dense model training on specific hardware, namely Ascend Neural Processing Units. The developers state that it achieves competitive performance against much larger models by leveraging efficient system optimizations and advanced normalization techniques during its creation.
How it came about
The model was introduced in a research paper submitted to arXiv in April 2025. The work represents an effort to train very large dense models efficiently. Previous advances in the field often focused on increasing scale through sparse architectures. This project instead focused on optimizing the training process for dense structures. The team utilized 8,192 Ascend NPUs to handle the computational load. The release highlights the maturity of the Ascend ecosystem for large-scale artificial intelligence workloads.
Inside the model
The architecture relies on dense Transformer modules. To manage the instability often found in training deep models, the team implemented depth-scaled sandwich normalization. This technique helps eliminate loss spikes, ensuring a smoother training curve. The model was pre-trained on a dataset comprising 13.2 trillion tokens, which were selected for diversity and quality. Following pre-training, the model underwent post-training phases specifically designed to enhance its reasoning capabilities. The entire process was supported by system optimizations tailored for Ascend hardware.
What it can do
Evaluations on various benchmarks suggest that Pangu Ultra advances the capabilities of dense large language models. The developers claim it performs comparably to Llama 405B and Mistral Large 2. It also achieves competitive results against DeepSeek-R1, a sparse model with significantly more parameters. These results indicate that dense models can remain competitive with sparse counterparts if trained effectively. The model shows particular strength in reasoning tasks due to its specialized post-training regimen.
How to use it
The model and its underlying system are not available as open weights. According to the authors, access is restricted to commercial customers. This suggests an application programming interface or private deployment model rather than public download. The focus is on demonstrating the viability of the hardware and software stack for enterprise use cases rather than broad community distribution.
Why it matters
Pangu Ultra demonstrates that Ascend NPUs can efficiently train dense models exceeding 100 billion parameters. This finding is significant for the hardware landscape, offering an alternative to graphics processing units from other vendors. It challenges the notion that sparse architectures are necessary for the largest models. The work provides a reference for system optimizations required for stable large-scale training on non-standard hardware accelerators.
Questions people ask about Pangu Ultra
Is Pangu Ultra an open-source model?
No, the model is not open-source. The developers state that the model and system will be available only to their commercial customers, implying restricted access via API or private license rather than public weight release.
What hardware was used to train Pangu Ultra?
The model was trained using 8,192 Ascend Neural Processing Units (NPUs). This demonstrates the capability of Huawei's Ascend hardware to handle large-scale dense model training efficiently with specific system optimizations.
How does Pangu Ultra compare to sparse models?
Despite being a dense model, it achieves competitive results with DeepSeek-R1, which is a sparse model with many more parameters. It also advances the state-of-the-art for dense models compared to Llama 405B and Mistral Large 2.
Good reading and sources
There is no English Wikipedia article on Pangu Ultra yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2504.07866. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.