News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
Switch Transformer

Switch Transformer

A sparse mixture-of-experts language model scaling to a trillion parameters.

Switch Transformer is a family of sparse language models built on T5, published in January 2021 by William Fedus, Barret Zoph and Noam Shazeer. It routes each input to a small set of parameters, keeping compute cost constant while growing the parameter count up to a trillion, and trains faster than dense T5 models.

Share on WhatsApp

  • Sparse mixture-of-experts design: huge parameter count, constant compute per input
  • Up to 7x faster pre-training than dense T5 at the same compute budget
  • Scaled to trillion-parameter models on the Colossal Clean Crawled Corpus
  • 4x speedup over T5-XXL reported at the largest scale
  • Gains over mT5-Base across all 101 languages tested

What it is

Switch Transformer is a sparse language model design introduced in a January 2021 paper by William Fedus, Barret Zoph and Noam Shazeer. It belongs to the T5 family and uses a mixture of experts approach, where different parameters handle different inputs instead of every input reusing the same weights. This gives a model a very large parameter count while keeping the computation per input roughly constant. The authors built versions based on T5-Base and T5-Large, and scaled the idea up to models with a trillion parameters, trained on the Colossal Clean Crawled Corpus.

How it came about

The work was posted on arXiv on 11 January 2021, with a revised version in June 2022. The authors set out to remove the barriers that had stopped mixture-of-experts models spreading: complicated routing, heavy communication between devices, and unstable training. Their answer was a simplified routing rule plus training methods that keep large sparse models stable. They showed the approach first on T5-Base and T5-Large, then extended it to multilingual mT5 and finally to trillion-parameter scale.

Inside the model

Ordinary dense models apply the same parameters to every input. Switch Transformer instead picks a small set of parameters for each example, so the model is sparsely activated: huge in total size, but cheap per token. The routing algorithm is simplified to cut communication and computation costs. The team also introduced training techniques to handle instability, and showed that sparse models of this kind can be trained in the low-precision bfloat16 number format, which saves memory and speed. Pre-training used the Colossal Clean Crawled Corpus.

What it can do

Compared with dense T5 models of the same computational budget, the Switch designs reached up to 7x faster pre-training. At the largest scale, the authors report a 4x speedup over T5-XXL. The gains also carried over to multilingual settings: measured against mT5-Base, improvements were recorded across all 101 languages tested.

How to use it

The work is published as an arXiv paper with a PDF available. The page lists links to code and media through services such as Hugging Face and CatalyzeX. No licence terms for the weights are given here.

Why it matters

Switch Transformer is presented as a step that made large sparse models practical: it simplified mixture-of-experts routing, lowered the cost of communication, and showed stable training at low precision. By reaching trillion-parameter scale with a 4x speedup over T5-XXL, it pushed the size of pre-trained language models further while keeping computation per input constant.

Questions people ask about Switch Transformer

What is a Switch Transformer?

It is a language model that uses a mixture of experts: instead of applying the same weights to every input, it selects a small set of parameters for each example. This keeps the computation per input low while allowing the total parameter count to grow very large.

Who created it and when?

William Fedus, Barret Zoph and Noam Shazeer published the work on arXiv on 11 January 2021, with a later revision in June 2022. The models are built on top of the T5 family.

How large can it get?

The authors pre-trained models up to a trillion parameters on the Colossal Clean Crawled Corpus. Because the model is sparsely activated, the compute cost per input stays roughly constant even as the parameter count grows.

How does it compare with T5?

Versions based on T5-Base and T5-Large reached up to 7x faster pre-training with the same computational resources. At the largest scale, the paper reports a 4x speedup over T5-XXL.

Does it work in other languages?

Yes. The improvements extended to multilingual settings, with gains measured over the mT5-Base version across all 101 languages in the study.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on Switch Transformer yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2101.03961. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.