News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
T0

T0

Encoder-decoder model for zero-shot task generalisation via prompts

T0 is a series of encoder-decoder language models developed by the BigScience Workshop. Released in 2021, it uses multitask prompted training to achieve strong zero-shot performance on English tasks, often outperforming much larger models like GPT-3 while being significantly smaller.

Share on WhatsApp

  • Outperforms GPT-3 on many tasks while being 16x smaller
  • Uses encoder-decoder architecture based on T5
  • Trained on P3 dataset with diverse natural language prompts
  • Available in 11 billion and 3 billion parameter sizes
  • Limited to English text and unable to process code

What it is

T0 is a family of encoder-decoder language models created by the BigScience Workshop. The primary variants contain 11 billion parameters, with a smaller 3 billion parameter version also available. These models are designed to perform natural language processing tasks without specific fine-tuning for each task, a capability known as zero-shot generalisation. By training on a diverse mixture of tasks described in natural language prompts, T0 can handle unseen queries by generating predictions based on the prompt structure. It is particularly noted for achieving high performance while being sixteen times smaller than comparable models like GPT-3.

How it came about

The model was introduced in a paper submitted in October 2021 and revised in March 2022. The work was presented as an ICLR 2022 Spotlight. The development team sought to test whether explicit multitask learning could induce zero-shot generalisation more effectively than implicit pretraining alone. They built upon the T5 architecture, using language model-adapted checkpoints that had undergone additional training steps. The project released all trained models and the prompt datasets, known as P3, to support reproducibility and further research in the community.

Inside the model

T0 uses an encoder-decoder transformer architecture derived from T5. The training process involves converting numerous supervised datasets into natural language prompts, with multiple templates for each task to vary the wording. This mixture covers areas such as question answering, sentiment analysis, summarisation, and paraphrase identification. The model is fine-tuned to autoregressively generate the target text given the prompted input. Training data sampling is proportional to the number of examples in each dataset, with packing used to combine examples into single sequences. The models were trained with bf16 activations, and developers recommend using bf16 or fp32 for inference rather than fp16.

What it can do

T0 demonstrates strong zero-shot performance on standard benchmarks, often outperforming models up to sixteen times its size. It excels in tasks such as natural language inference, coreference resolution, and word sense disambiguation. On a subset of the BIG-bench benchmark, it outperforms models six times its size. However, performance can vary depending on the specific prompt formulation used. The model is limited to English text and cannot process code or other languages due to tokenisation design choices. It requires significant computational resources to load and run, particularly the 11 billion parameter variants.

How to use it

The models are available with open weights through the Hugging Face Hub. Variants include T0, T0p, T0pp, and T0_3B. The T0pp checkpoint is recommended for best average performance across various tasks. Developers provide code examples for using the model with PyTorch and the Transformers library. Users are advised to replace the model path in the code if they wish to use a different variant. The associated training and evaluation data, the P3 dataset, is also publicly available for download.

Safety and testing

The developers excluded datasets with potentially harmful content from fine-tuning, but the models are not free from bias. Evaluations show that T0 can generate responses that are biased, offensive, or conspiratorial. For example, it has produced affirmative answers to questions about flat earth theories or vaccine misinformation. It also exhibits gender biases, such as associating men with architectural roles and women with nanny roles. In some instances, the model generated inappropriate sexual content when asked open-ended questions about innovation or social norms.

Why it matters

T0 has contributed to the understanding of how multitask prompted training affects zero-shot generalisation. By releasing the P3 dataset and model weights, it has enabled other researchers to benchmark their own models against a strong baseline. The approach demonstrates that explicit multitask learning can be more efficient than scaling model size alone, influencing subsequent work in prompt-based tuning and instruction following.

Questions people ask about T0

How do you pronounce the name T0?

The name should be pronounced as "T Zero". This references the concept of zero-shot learning. If you see variants like T0pp, the "p" stands for "Plus", so it is pronounced "T Zero Plus Plus".

Can T0 process code or non-English languages?

No. Due to specific design choices in its tokenisation process, T0 is unable to perform inference for tasks involving computer code or any text that is not in English. It is strictly an English-language model.

Which variant of T0 should I use for best results?

The developers recommend using the T0pp checkpoint. This version includes additional datasets from SuperGLUE and generally leads to the best average performance across a wide variety of natural language processing tasks.

What hardware is needed to run T0?

The models are quite large, with 11 billion or 3 billion parameters. Running them requires non-trivial computational resources. If you have multiple GPUs, you can use parallelisation techniques to distribute the load and make inference feasible.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on T0 yet, so the facts on this page come from the sources listed above: huggingface.co/bigscience/T0, arxiv.org/abs/2110.08207. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.