News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
Pangu Ultra

Pangu Ultra

A 135-billion parameter dense language model trained on Ascend hardware.

Pangu Ultra is a large language model with 135 billion parameters, developed by Huawei researchers. Released in April 2025, it was trained on Ascend Neural Processing Units to demonstrate the capability of dense architectures at scale. The model aims to compete with larger sparse models through optimized training techniques and extensive post-training for reasoning tasks.

Share on WhatsApp

  • Contains 135 billion parameters in a dense structure.
  • Trained on 13.2 trillion high-quality tokens.
  • Uses 8,192 Ascend NPUs for efficient training.
  • Employs depth-scaled sandwich normalization to stabilize training.
  • Available exclusively to commercial customers.

What it is

Pangu Ultra is a dense large language model containing 135 billion parameters. It was created by a team at Huawei and described in a paper submitted in April 2025. The model uses standard Transformer modules rather than sparse mixture-of-experts structures. Its primary purpose is to push the limits of dense model training on specific hardware, namely Ascend Neural Processing Units. The developers state that it achieves competitive performance against much larger models by leveraging efficient system optimizations and advanced normalization techniques during its creation.

How it came about

The model was introduced in a research paper submitted to arXiv in April 2025. The work represents an effort to train very large dense models efficiently. Previous advances in the field often focused on increasing scale through sparse architectures. This project instead focused on optimizing the training process for dense structures. The team utilized 8,192 Ascend NPUs to handle the computational load. The release highlights the maturity of the Ascend ecosystem for large-scale artificial intelligence workloads.

Inside the model

The architecture relies on dense Transformer modules. To manage the instability often found in training deep models, the team implemented depth-scaled sandwich normalization. This technique helps eliminate loss spikes, ensuring a smoother training curve. The model was pre-trained on a dataset comprising 13.2 trillion tokens, which were selected for diversity and quality. Following pre-training, the model underwent post-training phases specifically designed to enhance its reasoning capabilities. The entire process was supported by system optimizations tailored for Ascend hardware.

What it can do

Evaluations on various benchmarks suggest that Pangu Ultra advances the capabilities of dense large language models. The developers claim it performs comparably to Llama 405B and Mistral Large 2. It also achieves competitive results against DeepSeek-R1, a sparse model with significantly more parameters. These results indicate that dense models can remain competitive with sparse counterparts if trained effectively. The model shows particular strength in reasoning tasks due to its specialized post-training regimen.

How to use it

The model and its underlying system are not available as open weights. According to the authors, access is restricted to commercial customers. This suggests an application programming interface or private deployment model rather than public download. The focus is on demonstrating the viability of the hardware and software stack for enterprise use cases rather than broad community distribution.

Why it matters

Pangu Ultra demonstrates that Ascend NPUs can efficiently train dense models exceeding 100 billion parameters. This finding is significant for the hardware landscape, offering an alternative to graphics processing units from other vendors. It challenges the notion that sparse architectures are necessary for the largest models. The work provides a reference for system optimizations required for stable large-scale training on non-standard hardware accelerators.

Questions people ask about Pangu Ultra

Is Pangu Ultra an open-source model?

No, the model is not open-source. The developers state that the model and system will be available only to their commercial customers, implying restricted access via API or private license rather than public weight release.

What hardware was used to train Pangu Ultra?

The model was trained using 8,192 Ascend Neural Processing Units (NPUs). This demonstrates the capability of Huawei's Ascend hardware to handle large-scale dense model training efficiently with specific system optimizations.

How does Pangu Ultra compare to sparse models?

Despite being a dense model, it achieves competitive results with DeepSeek-R1, which is a sparse model with many more parameters. It also advances the state-of-the-art for dense models compared to Llama 405B and Mistral Large 2.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on Pangu Ultra yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2504.07866. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.