T5
Encoder-decoder language model family from Google AI using a text-to-text framework.
Google AI introduced the T5 series of large language models in October 2019. These encoder-decoder transformers convert all tasks into a unified text format, enabling broad applications through pre-training and fine-tuning.
- Released by Google AI in October 2019 under Apache-2.0 licence.
- Uses an encoder-decoder transformer trained on the C4 dataset.
- Converts all tasks into a uniform text-to-text format.
- Includes variants like Switch Transformer and Flan-T5.
- Encoder used as text conditioner for diffusion image models.
What it is
T5 is a family of large language models created by Google AI and released in late 2019. It treats every problem as a text-to-text conversion, meaning inputs and outputs are always strings. The system uses an encoder to read data and a decoder to write responses. This design allows the same architecture to handle diverse tasks like translation or summarization without changing the underlying structure.
How it came about
The original models launched in October 2019 with sizes ranging from small to 11 billion parameters. Subsequent updates improved performance, such as the 1.1 series which switched activation functions. Later variants included LM-adapted T5 for better language modeling and Switch Transformer, which introduced mixture-of-experts layers. Instruction-tuned versions like Flan-T5 appeared in 2022 to improve following specific prompts.
Inside the model
The architecture relies on an encoder-decoder transformer structure where both parts have equal layer counts. Training used the C4 dataset, a massive collection of internet text and code. A shared WordPiece tokenizer with a 32,000-word vocabulary processed inputs. Modifications included relative positional embeddings and bias-free layer normalization. All tasks were formatted as input text followed by output text during pre-training.
What it can do
Models can perform machine translation, text summarization, and grammatical judgment after fine-tuning. They handle corrupted text restoration and code generation. The encoder also serves as a standalone text feature extractor for other systems. Variants like ByT5 operate directly on byte sequences rather than tokens, supporting multilingual processing across different languages.
How to use it
The weights are available under the Apache-2.0 licence, allowing open use. The original implementation was written in TensorFlow. A later JAX-based re-implementation called T5X provides updated tools. Researchers can download checkpoints for various sizes, from small base models to larger XXL versions, for local experimentation.
Safety and testing
The provided materials do not describe specific safety evaluations, red-teaming exercises, or content filtering mechanisms.
Reception and debate
The model is noted for its architectural influence and adoption in research projects and subsequent derivative systems.
Why it matters
T5 established a standardized text-to-text approach that influenced many downstream models. Its encoder is used in image generation systems like Google Imagen and AuraFlow as a text conditioning component. The architecture inspired variations like UL2 and Pile-T5, demonstrating its flexibility for scaling and different training objectives.
Questions people ask about T5
What is the main difference between T5 and other models?
T5 converts every task into a text-to-text format. Instead of separate heads for classification or regression, it generates text outputs for all problems, simplifying the architecture.
How does T5 process input text?
It uses a shared WordPiece tokenizer with a 32,000 word vocabulary. The encoder processes these tokens while the decoder generates the output sequence autoregressively.
Can T5 be used for image generation?
Yes, the T5 encoder acts as a text encoder for diffusion models. Systems like Google Imagen use T5-XXL to encode text instructions that condition the image generation process.
What datasets train T5 models?
The primary dataset is C4, a colossal clean crawled corpus of internet text and code. Some variants use multilingual data or instruction datasets like FLAN for fine-tuning.
Good reading and sources
- google/flan-t5-xl · Hugging FaceHugging Face, Article
- The Power of Scale for Parameter-Efficient Prompt TuningarXiv, Article
The facts on this page come from the Wikipedia article "T5 (language model)", written by its contributors and used under the CC BY-SA 4.0 licence, and from its Wikidata record (CC0). EveryLLM rewrote the text in its own words, with AI assistance, and compares every page with the article to make sure no passage is copied. It may be reused under the same CC BY-SA 4.0 licence with a link back.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-15.