Megatron-LM
Model-parallel training techniques for transformer language models with billions of parameters
Megatron-LM is a set of techniques and code for training very large transformer language models, presented by NVIDIA researchers in a paper first posted in September 2019. It splits each layer across multiple GPUs so models of billions of parameters can be trained without a new compiler or changes to libraries, and demonstrated GPT-2- and BERT-style models up to 8.3 billion parameters.
- Trains transformer models up to 8.3 billion parameters on 512 GPUs
- Intra-layer model parallelism added to native PyTorch with a few communication ops
- Sustains 15.1 PetaFLOPs at 76% scaling efficiency versus a single-GPU baseline
- GPT-2-style model: WikiText103 perplexity 10.8, LAMBADA accuracy 66.5%
- BERT-style model: 90.9% accuracy on RACE
What it is
Megatron-LM is a method for training transformer language models that are too large to fit on a single GPU. The name refers both to the paper and the accompanying code. Its authors are Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper and Bryan Catanzaro. The work was first posted in September 2019 and revised through March 2020. Using the approach, the team trained a GPT-2-like model with 8.3 billion parameters and a BERT-like model with 3.9 billion parameters. The goal was to show that scaling up model size improves results on natural language tasks, and to make such training practical on ordinary GPU clusters.
How it came about
The paper appeared on arXiv on 17 September 2019, with a fourth revision dated 13 March 2020. It was written by six researchers. The work set out to solve a memory problem: as transformer models grow, they no longer fit in the memory of one accelerator. The team built a simple way to spread a model across devices and showed it working on models far larger than what a single GPU could hold.
Inside the model
The core idea is intra-layer model parallelism. Each layer of the transformer is divided across several GPUs, with a small number of communication operations inserted into standard PyTorch code. No new compiler or library changes are needed. The technique works alongside pipeline parallelism, which splits the model by stages, so the two can be combined. The authors trained models up to 8.3 billion parameters on 512 GPUs, sustaining 15.1 PetaFLOPs across the run, which they report as 76% scaling efficiency against a strong single-GPU baseline of 39 TeraFLOPs, itself 30% of peak. For BERT-like models they found that where layer normalisation is placed matters a great deal as size grows.
What it can do
The 8.3 billion parameter GPT-2-style model reached a perplexity of 10.8 on WikiText103, against a reported state of the art of 15.8, and 66.5% accuracy on LAMBADA against 63.2%. The BERT-style model scored 90.9% on RACE, compared with a reported state of the art of 89.4%. These figures are the ones given in the paper's abstract.
How to use it
Megatron-LM is implemented in native PyTorch and the paper describes it as needing only a few communication operations added to existing code. The abstract presents the approach as fully implementable without a new compiler or library changes.
Why it matters
Megatron-LM showed that model parallelism could be done inside ordinary PyTorch, without special tooling, and used it to push transformer language models into the multi-billion parameter range. The paper demonstrated that larger GPT-2- and BERT-style models improved results on standard benchmarks, and its scaling figures gave a reference point for training such models across hundreds of GPUs.
Questions people ask about Megatron-LM
What is Megatron-LM?
It is a technique and codebase for training very large transformer language models by splitting each layer across multiple GPUs. The authors are NVIDIA researchers, and the work was first posted in September 2019. It demonstrated GPT-2- and BERT-style models with billions of parameters.
How does the parallelism work?
It divides individual layers across devices, inserting a few communication operations into standard PyTorch code. The authors say it needs no new compiler and no changes to libraries. It can also be combined with pipeline parallelism, which splits the model into stages, since the two methods are complementary.
How large a model did it train?
The largest model in the paper has 8.3 billion parameters, a GPT-2-like transformer. A BERT-like model with 3.9 billion parameters was also trained. Both were run on 512 GPUs.
What results did it report?
The GPT-2-style model reached perplexity 10.8 on WikiText103 and 66.5% accuracy on LAMBADA. The BERT-style model reached 90.9% on RACE. The paper compares each figure with the best result it knew of at the time.
Good reading and sources
There is no English Wikipedia article on Megatron-LM yet, so the facts on this page come from the sources listed above: arxiv.org/abs/1909.08053. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.