Trillion-scale: 1 trillion and more
Kimi K2, the Switch Transformer and the largest models whose size is published. 19 models.
WorldSwitch Transformer
A sparse mixture-of-experts language model scaling to a trillion parameters.Research · Research only · 2021
🇺🇸 United StatesGLaM
A trillion-parameter mixture-of-experts language model from Google ResearchResearch · Research only · 2021
🇨🇳 ChinaPanGu-Sigma
Trillion-parameter sparse language model for Chinese NLP tasksGeneral · Research only · 2023