Cerebras-GPT
Open compute-optimal language models from Cerebras Systems
Cerebras-GPT is a family of open large language models released by Cerebras Systems in April 2023. Ranging from 111 million to 13 billion parameters, these models follow Chinchilla scaling laws and are trained on the Pile dataset using Cerebras wafer-scale hardware.
- Family ranges from 111M to 13B parameters
- Trained on 371 billion tokens from The Pile
- Follows Chinchilla compute-optimal scaling rules
- Released under permissive Apache License
- Trained on Cerebras Andromeda supercomputer
What it is
Cerebras-GPT is a series of transformer-based language models developed by Cerebras Systems. Released in April 2023, the family includes seven sizes ranging from 111 million to 13 billion parameters. The models are designed to demonstrate efficient scaling on Cerebras hardware while adhering to open data and architecture standards. They serve as reference implementations for researchers studying large model training dynamics.
How it came about
The project was announced in April 2023 alongside an arXiv paper detailing the training process. Cerebras Systems built the models to validate their wafer-scale cluster capabilities. The team trained the models sequentially from smallest to largest, using power-law extrapolation to predict validation loss for subsequent sizes. This approach allowed them to monitor training health and ensure compute efficiency across the entire family.
Inside the model
The architecture follows a GPT-3 style dense decoder design with full attention layers. Models were trained on the Andromeda supercomputer, which uses Cerebras CS-2 wafer-scale systems. Training utilized the Pile dataset, comprising 371 billion tokens processed with byte-pair encoding. The team applied Chinchilla scaling rules, targeting twenty tokens per parameter for compute optimisation. Maximal Update Parameterization was used to improve hyperparameter predictability at scale.
What it can do
Evaluations used the Eleuther lm-evaluation-harness on tasks such as HellaSwag, PIQA, and Lambada. The 13 billion parameter model achieved a downstream average score of 0.570 in zero-shot settings. Performance improved consistently with size, showing predictable scaling behaviour. The models demonstrated state-of-the-art training efficiency compared to other publicly available counterparts during their release period.
How to use it
All models are hosted on Hugging Face under a permissive Apache License. Weights and code are fully open for community use, fine-tuning, and adaptation. Researchers can access pre-trained checkpoints via the Cerebras Model Zoo. The company also offers cloud-based training and fine-tuning services through the Cerebras Model Studio for those requiring additional compute resources.
Why it matters
The release provided a reproducible benchmark for compute-optimal scaling laws. By sharing detailed training configurations and hyperparameters, Cerebras enabled other researchers to replicate results without proprietary barriers. The work highlighted the viability of wafer-scale engines for training large dense models efficiently, influencing discussions on hardware-software co-design in artificial intelligence.
Questions people ask about Cerebras-GPT
What dataset was used to train Cerebras-GPT?
The models were trained on the Pile dataset from EleutherAI. This corpus contains 371 billion tokens after cleaning and tokenization using byte-pair encoding with the GPT-2 vocabulary.
Are the Cerebras-GPT models open source?
Yes, all models in the family are released with open weights under a permissive Apache License. The code and training details are also available to facilitate reproducibility and community research.
How many parameters does the largest Cerebras-GPT model have?
The largest model in the family has 13 billion parameters. It consists of 40 layers and was trained on 257 billion tokens to meet the Chinchilla scaling target of twenty tokens per parameter.
Good reading and sources
There is no English Wikipedia article on Cerebras-GPT yet, so the facts on this page come from the sources listed above: huggingface.co/cerebras-archive/Cerebras-GPT-13B, arxiv.org/abs/2304.03208. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.