News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
OpenELM

OpenELM

Apple's small open language models with a full training and inference release

OpenELM is a family of compact language models released by Apple in April 2024, spanning 270 million to 3 billion parameters in both base and instruction-tuned forms. Apple published the weights, training recipes, logs and evaluation code, aiming to make language model research easier to reproduce and inspect.

Share on WhatsApp

  • Four sizes: 270M, 450M, 1.1B and 3B parameters, base and instruction-tuned
  • Pretrained on roughly 1.8 trillion tokens of public datasets
  • Layer-wise parameter scaling claims 2.36% better accuracy than OLMo at ~1B
  • Uses half the pre-training tokens of OLMo at that size, the authors say
  • Weights, logs, checkpoints and training code all published

What it is

OpenELM is a set of small language models published by Apple in 2024. Four sizes are offered: 270 million, 450 million, 1.1 billion and 3 billion parameters, and each size comes in a plain pretrained form and an instruction-following form. A parameter is one learned number inside the model. The design spreads parameters unevenly across the layers of the transformer, the standard neural network for text, so that accuracy improves for a given size. Apple says it wanted to make language model work easier to reproduce and to examine for bias and risk.

How it came about

Apple researchers presented OpenELM in a technical paper first posted on 22 April 2024, with a revised version on 2 May 2024. Alongside the paper, the team put the model weights on Hugging Face and published the full pipeline: data preparation, training, fine-tuning and evaluation, plus training logs and several checkpoints. The models were pretrained with Apple's CoreNet library. Code was also supplied for converting the models to MLX, Apple's library for running and fine-tuning models on its own devices.

Inside the model

The transformer in OpenELM assigns parameters layer by layer rather than giving every layer the same width, which the authors say buys accuracy for the same parameter budget. At roughly one billion parameters they report a 2.36% accuracy gain over OLMo while using half as many pre-training tokens. Pre-training drew on RefinedWeb, a deduplicated version of PILE, and subsets of RedPajama and Dolma v1.6, adding up to about 1.8 trillion tokens. Tokenisation uses the LLaMA tokenizer, which needs a beginning-of-sequence token added. Both raw and instruction-tuned checkpoints are published.

What it can do

Reported zero-shot averages across seven tasks rise from 54.37 for the 270M base model to 69.15 for the 3B instruction-tuned one; the instruction versions generally score above their base counterparts. On the LLM360 and OpenLLM Leaderboard task sets the same pattern holds, with the 3B-Instruct model averaging 51.22 and 55.73 respectively. MMLU scores stay low across the family, around 25 to 27. Results are given as published, without ranking against other models here.

How to use it

All eight checkpoints are downloadable from Hugging Face under the Apple organisation. The accompanying repository covers data preparation, training, fine-tuning and evaluation, and includes training logs and multiple checkpoints. A sample script shows how to generate text, with options such as a repetition penalty, prompt-lookup speculative decoding and a smaller assistant model for faster output. Conversion code targets MLX for use on Apple hardware.

Safety and testing

The authors frame the release partly around openness to risk and bias research, arguing that public weights, data recipes and logs let others probe these issues. The published evaluations include CrowS-Pairs, a benchmark for social bias, alongside standard accuracy tasks. No separate red-teaming or incident reports are described in these pages.

Reception and debate

The pages available here are the project's own model card and paper listing, so they report Apple's claims rather than outside assessment. The authors position the release against prior practice of shipping weights and inference code only, while training on private data. No reviews, lawsuits or bans are described here.

Why it matters

By shipping the whole pipeline rather than just weights, OpenELM is presented as a resource for reproducible language model research. The layer-wise scaling approach is offered as a way to get more accuracy from a fixed parameter budget, illustrated against OLMo at the one-billion-parameter scale. The MLX conversion code ties the models to on-device use on Apple products.

Questions people ask about OpenELM

What is OpenELM?

A family of small language models released by Apple in April 2024. It comes in four sizes, from 270 million to 3 billion parameters, each with a pretrained and an instruction-tuned version. Apple published the weights together with the training and evaluation framework.

What does the name stand for?

The letters expand to Open Efficient Language Models. The 'open' part refers to the release of weights, training recipes, logs and code, not just the final model files.

What data was it trained on?

Roughly 1.8 trillion tokens drawn from RefinedWeb, a deduplicated PILE, a subset of RedPajama and a subset of Dolma v1.6. Apple notes that users should check the licence terms of those datasets themselves.

How do I run it?

Weights are on Hugging Face, and a sample script covers text generation with options like repetition penalty and speculative decoding. There is also code to convert the models to MLX for inference and fine-tuning on Apple devices.

How large is the biggest version?

The largest is OpenELM-3B, with 3 billion parameters. Its instruction-tuned variant averages 69.15 across seven zero-shot tasks as reported in the model card.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on OpenELM yet, so the facts on this page come from the sources listed above: huggingface.co/apple/OpenELM-3B-Instruct, arxiv.org/abs/2404.14619. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.