News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
Megatron-Turing NLG 530B

Megatron-Turing NLG 530B

A 530-billion-parameter generative language model trained by Microsoft and NVIDIA.

Megatron-Turing NLG 530B (MT-NLG) is a general-purpose generative language model with 530 billion parameters, built jointly by Microsoft and NVIDIA. Described in a January 2022 paper, it was presented as the largest monolithic transformer-based language model of its kind and reported strong zero-, one- and few-shot results across NLP benchmarks.

Share on WhatsApp

  • 530 billion parameters, described as the largest monolithic transformer language model
  • Joint work between Microsoft and NVIDIA
  • Trained using DeepSpeed and Megatron with 3D parallelism
  • Paper first posted 28 January 2022 on arXiv
  • Reported new state-of-the-art zero-, one- and few-shot results

What it is

Megatron-Turing NLG 530B, shortened to MT-NLG, is a large generative language model with 530 billion parameters. It came out of a joint effort between Microsoft and NVIDIA and was described in a paper first posted in January 2022. The authors call it the largest monolithic transformer-based language model, meaning one built as a single model rather than split into separate pieces. It is a general-purpose system meant to be adapted to many natural language processing tasks through zero-shot, one-shot, few-shot prompting or fine-tuning.

How it came about

The work was carried out by a team of twenty authors from Microsoft and NVIDIA, with Shaden Smith and Mostofa Patwary listed as equal contributors. The paper appeared on arXiv on 28 January 2022 and was revised twice, with the latest version dated 4 February 2022. It set out the training infrastructure, the data work and the evaluation results for the model.

Inside the model

Training a model of this size needed both specialised software and a way to spread the work across many machines. The team used DeepSpeed and Megatron together, applying what they call 3D parallelism to divide the model and the computation. The paper also describes how the training corpus was assembled and cleaned, and the authors argue that careful data curation was a key reason the model worked well. Beyond these points, the extract gives no layer counts, context length or training compute figures.

What it can do

The authors report that MT-NLG reached strong zero-, one- and few-shot accuracies on several natural language processing benchmarks, and say it set new state-of-the-art results at the time. They also note new behaviours and properties observed in the model during the work. Specific benchmark names and scores are not given in this extract.

How to use it

The extract describes a research paper rather than a released product. No downloadable weights, licence terms, app or API are described here.

Why it matters

The authors present the project as a contribution to large-scale training infrastructure, to large language models and to natural language generation more broadly. They say the details they publish should help others working on training models at this scale.

Questions people ask about Megatron-Turing NLG 530B

What is Megatron-Turing NLG 530B?

It is a general-purpose generative language model with 530 billion parameters, built jointly by Microsoft and NVIDIA. The team describes it as the largest monolithic transformer-based language model, meaning a single model rather than several linked ones, and it is designed to be adapted to many language tasks.

Who made it and when?

Microsoft and NVIDIA developed it together. A paper describing the model was first posted on arXiv on 28 January 2022, with a later revision on 4 February 2022. Twenty authors are listed, with Shaden Smith and Mostofa Patwary credited as equal contributors.

How was such a large model trained?

The team used two software systems, DeepSpeed and Megatron, together with a method they call 3D parallelism to spread training across hardware. They also put effort into building and cleaning the training corpus, which they say was a key ingredient in the model's success.

How well does it perform?

The authors report strong zero-, one- and few-shot accuracies on several natural language processing benchmarks, and say the model set new state-of-the-art results at the time. The extract does not list individual benchmark names or scores.

Can I use or download it?

The available description covers a research paper only. It does not describe downloadable weights, a licence, an app or an API for the model.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on Megatron-Turing NLG 530B yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2201.11990. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.