OLMoE
Open mixture-of-experts language model with 1 billion active parameters
OLMoE is a fully open large language model released by the Allen Institute for AI in September 2024. It uses a sparse mixture-of-experts architecture to deliver high performance with low computational cost, providing complete transparency through open weights, data and code.
- 7 billion total parameters with 1 billion active per token
- Pretrained on 5 trillion tokens of open data
- Outperforms Llama2-13B-Chat on several benchmarks
- Fully open weights, code, data and training logs
- Uses sparse Mixture-of-Experts architecture
What it is
OLMoE is a large language model developed by the Allen Institute for AI. Released in September 2024, it features a total of seven billion parameters but activates only one billion for each input token. This design allows it to run efficiently while maintaining strong performance. The project is notable for its complete openness, releasing all model weights, training datasets, code and training logs to the public. It serves as a resource for researchers seeking to study and build upon transparent artificial intelligence systems.
How it came about
The model was introduced in a paper submitted to arXiv in September 2024 by a team including Niklas Muennighoff and Hannaneh Hajishirzi. It builds upon the earlier OLMo series, aiming to improve efficiency through sparse architectures. The developers released pretraining checkpoints, supervised fine-tuning versions and instruction-tuned variants. All components, including the data mix and optimization code, were made available simultaneously with the model announcement to support reproducibility.
Inside the model
OLMoE uses a sparse Mixture-of-Experts architecture. This means the model contains multiple specialized sub-networks, or experts, but only routes a small fraction of them to process any given piece of text. It was pretrained on five trillion tokens of data. The training process included supervised fine-tuning and direct preference optimization to align the model with human instructions. The developers stored weights in BF16 format for efficiency, though FP32 versions are also available. Routing analysis showed high specialization among the experts.
What it can do
The model achieves state-of-the-art results among systems with similar active parameter counts. It scores 54.1 on the MMLU benchmark and 80.0 on HellaSwag. These figures surpass other open models with around one billion active parameters, such as TinyLlama and Pythia. It also competes with larger dense models, outperforming Llama2-13B-Chat in several metrics. The instruction-tuned variant demonstrates strong capabilities in following complex directions while maintaining efficiency.
How to use it
All aspects of OLMoE are open source. Users can download the model weights from Hugging Face. The training data, code for pretraining and fine-tuning, and detailed training logs are publicly accessible. To use the model, developers must install specific versions of the transformers library from source. The licence permits full usage and modification, supporting both academic research and commercial application without restrictive barriers.
Why it matters
OLMoE sets a new standard for transparency in large language model development. By releasing every component of the training pipeline, it enables other researchers to verify results and build upon its methods. Its success demonstrates that sparse architectures can deliver high performance at lower computational costs, encouraging further exploration of mixture-of-experts designs in the open-source community.
Questions people ask about OLMoE
What is the difference between total and active parameters?
OLMoE has seven billion total parameters but uses a mixture-of-experts design. This means only one billion parameters are active for any single input token. This approach reduces computational cost during inference while retaining the knowledge capacity of a larger model.
Is OLMoE free to use?
Yes, the model is fully open source. You can download the weights, training data and code from Hugging Face and GitHub. There are no restrictions mentioned in the provided sources regarding its use for research or commercial purposes.
How does OLMoE compare to Llama models?
Despite having fewer active parameters, OLMoE outperforms Llama2-13B-Chat on several benchmarks. It achieves higher scores on MMLU and HellaSwag than many larger dense models, showing that its sparse architecture is highly efficient.
Good reading and sources
There is no English Wikipedia article on OLMoE yet, so the facts on this page come from the sources listed above: huggingface.co/allenai/OLMoE-1B-7B-0924, arxiv.org/abs/2409.02060. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.