Turing-NLG
Microsoft's 17-billion-parameter Transformer model for generation, QA and summarisation
Turing-NLG (T-NLG) is a 17-billion-parameter generative language model from Microsoft, presented in 2020 as the largest model published at that time. Built on a Transformer design and trained with DeepSpeed and ZeRO, it writes direct answers and abstractive summaries rather than copying text from source documents.
- 17 billion parameters, described by Microsoft as the largest model published at the time
- 78 Transformer layers, hidden size 4256, 28 attention heads
- Trained with DeepSpeed and ZeRO on 256 NVIDIA GPUs at batch size 512
- Zero-shot question answering with no supporting passage
- Private demo only, shared with a small group of academics
What it is
Turing Natural Language Generation, shortened to T-NLG, is a generative language model with 17 billion parameters, built by Microsoft under the Project Turing banner. It is a Transformer-based system, which means it produces words to finish open-ended text tasks rather than selecting passages from a document. Microsoft says it handles freeform generation, question answering and summarisation, and that it beat the state of the art on several language modelling benchmarks at the time of publication. The company positioned it as the largest model published up to that point and showed it to a small group of academics for feedback.
How it came about
Microsoft Project Turing introduced T-NLG as part of a trend towards ever larger language models. The team released a private demo covering freeform generation, question answering and summarisation to a limited set of academic users, aiming to gather initial testing and feedback. The work leaned on the DeepSpeed library and the ZeRO optimizer, both compatible with PyTorch, which the team described as essential to training a model of this size.
Inside the model
T-NLG has 78 Transformer layers, a hidden size of 4256 and 28 attention heads. Because anything above about 1.3 billion parameters will not fit on a single 32GB GPU, the model was split across hardware: tensor slicing sharded it over four NVIDIA V100 GPUs on the Megatron-LM framework, running on NVIDIA DGX-2 machines linked by InfiniBand. DeepSpeed with ZeRO cut the model-parallelism degree from 16 to 4, raised per-node batch size fourfold and shortened training roughly threefold. Pretraining used autoregressive generation loss for 300,000 steps at batch size 512 on 1,024-token sequences, with 3,200 warmup steps to a peak learning rate of 1.5x10-4, cosine decay over 500,000 steps and FP16 precision.
What it can do
Microsoft reported new state-of-the-art results on LAMBADA next-word prediction accuracy and WikiText-103 perplexity, comparing against the publicly released Megatron-LM numbers. The model answers questions directly in complete sentences instead of surfacing a highlighted passage, and can do zero-shot question answering with no supporting context, drawing on knowledge from pretraining. Human annotators judged factual and grammatical quality against an LSTM baseline similar to CopyNet, since ROUGE scores alone do not capture those aspects. The team noted that a larger pretrained model needs fewer downstream examples, reaching the baseline's level after only a few thousand training instances.
How to use it
Access was limited to a private demo shown to a small set of academic users, covering freeform generation, question answering and summarisation. No public weights or licence were released.
Why it matters
Microsoft framed T-NLG as evidence that scale and diverse pretraining data improve generalisation across tasks, arguing it is more efficient to train one large multi-task model than a separate model per task. The training work also fed into DeepSpeed and ZeRO, tools for fitting very large models onto fewer GPUs.
Questions people ask about Turing-NLG
What is Turing-NLG?
It is a 17-billion-parameter generative language model from Microsoft, part of Project Turing. It is Transformer-based and produces text for open-ended tasks such as writing freeform answers, replying to questions and summarising documents, rather than copying sentences out of them.
How was such a large model trained?
The model was split across multiple GPUs using tensor slicing on NVIDIA Megatron-LM, running on NVIDIA DGX-2 hardware with InfiniBand links. DeepSpeed with ZeRO reduced the parallelism needed from 16 GPUs to 4 and let the team train at batch size 512 on 256 GPUs instead of 1,024.
How can I use it?
Microsoft offered a private demo to a small set of academic users, covering freeform generation, question answering and summarisation. The company described the purpose as collecting initial testing and feedback; no public release of weights is described.
What can it do that earlier systems could not?
Earlier question answering and summarisation systems pulled existing sentences from documents, which often read unnaturally. T-NLG writes its own answers and abstractive summaries, and can answer questions with no supporting passage at all, relying on what it learned during pretraining.
Good reading and sources
There is no English Wikipedia article on Turing-NLG yet, so the facts on this page come from the sources listed above: microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-la. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.