News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
OLMoE

OLMoE

Open mixture-of-experts language model with 1 billion active parameters

OLMoE is a fully open large language model released by the Allen Institute for AI in September 2024. It uses a sparse mixture-of-experts architecture to deliver high performance with low computational cost, providing complete transparency through open weights, data and code.

Share on WhatsApp

  • 7 billion total parameters with 1 billion active per token
  • Pretrained on 5 trillion tokens of open data
  • Outperforms Llama2-13B-Chat on several benchmarks
  • Fully open weights, code, data and training logs
  • Uses sparse Mixture-of-Experts architecture

What it is

OLMoE is a large language model developed by the Allen Institute for AI. Released in September 2024, it features a total of seven billion parameters but activates only one billion for each input token. This design allows it to run efficiently while maintaining strong performance. The project is notable for its complete openness, releasing all model weights, training datasets, code and training logs to the public. It serves as a resource for researchers seeking to study and build upon transparent artificial intelligence systems.

How it came about

The model was introduced in a paper submitted to arXiv in September 2024 by a team including Niklas Muennighoff and Hannaneh Hajishirzi. It builds upon the earlier OLMo series, aiming to improve efficiency through sparse architectures. The developers released pretraining checkpoints, supervised fine-tuning versions and instruction-tuned variants. All components, including the data mix and optimization code, were made available simultaneously with the model announcement to support reproducibility.

Inside the model

OLMoE uses a sparse Mixture-of-Experts architecture. This means the model contains multiple specialized sub-networks, or experts, but only routes a small fraction of them to process any given piece of text. It was pretrained on five trillion tokens of data. The training process included supervised fine-tuning and direct preference optimization to align the model with human instructions. The developers stored weights in BF16 format for efficiency, though FP32 versions are also available. Routing analysis showed high specialization among the experts.

What it can do

The model achieves state-of-the-art results among systems with similar active parameter counts. It scores 54.1 on the MMLU benchmark and 80.0 on HellaSwag. These figures surpass other open models with around one billion active parameters, such as TinyLlama and Pythia. It also competes with larger dense models, outperforming Llama2-13B-Chat in several metrics. The instruction-tuned variant demonstrates strong capabilities in following complex directions while maintaining efficiency.

How to use it

All aspects of OLMoE are open source. Users can download the model weights from Hugging Face. The training data, code for pretraining and fine-tuning, and detailed training logs are publicly accessible. To use the model, developers must install specific versions of the transformers library from source. The licence permits full usage and modification, supporting both academic research and commercial application without restrictive barriers.

Why it matters

OLMoE sets a new standard for transparency in large language model development. By releasing every component of the training pipeline, it enables other researchers to verify results and build upon its methods. Its success demonstrates that sparse architectures can deliver high performance at lower computational costs, encouraging further exploration of mixture-of-experts designs in the open-source community.

Questions people ask about OLMoE

What is the difference between total and active parameters?

OLMoE has seven billion total parameters but uses a mixture-of-experts design. This means only one billion parameters are active for any single input token. This approach reduces computational cost during inference while retaining the knowledge capacity of a larger model.

Is OLMoE free to use?

Yes, the model is fully open source. You can download the weights, training data and code from Hugging Face and GitHub. There are no restrictions mentioned in the provided sources regarding its use for research or commercial purposes.

How does OLMoE compare to Llama models?

Despite having fewer active parameters, OLMoE outperforms Llama2-13B-Chat on several benchmarks. It achieves higher scores on MMLU and HellaSwag than many larger dense models, showing that its sparse architecture is highly efficient.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on OLMoE yet, so the facts on this page come from the sources listed above: huggingface.co/allenai/OLMoE-1B-7B-0924, arxiv.org/abs/2409.02060. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.