Trillion-scale: 1 trillion and more
Kimi K2, the Switch Transformer and the largest models whose size is published. 19 models.
WorldSwitch Transformer
A sparse mixture-of-experts language model scaling to a trillion parameters.Research 路 Research only 路 2021
馃嚭馃嚫 United StatesGLaM
A trillion-parameter mixture-of-experts language model from Google ResearchResearch 路 Research only 路 2021
馃嚚馃嚦 ChinaPanGu-Sigma
Trillion-parameter sparse language model for Chinese NLP tasksGeneral 路 Research only 路 2023