OpenELM
Apple's small open language models with a full training and inference release
OpenELM is a family of compact language models released by Apple in April 2024, spanning 270 million to 3 billion parameters in both base and instruction-tuned forms. Apple published the weights, training recipes, logs and evaluation code, aiming to make language model research easier to reproduce and inspect.
- Four sizes: 270M, 450M, 1.1B and 3B parameters, base and instruction-tuned
- Pretrained on roughly 1.8 trillion tokens of public datasets
- Layer-wise parameter scaling claims 2.36% better accuracy than OLMo at ~1B
- Uses half the pre-training tokens of OLMo at that size, the authors say
- Weights, logs, checkpoints and training code all published
What it is
OpenELM is a set of small language models published by Apple in 2024. Four sizes are offered: 270 million, 450 million, 1.1 billion and 3 billion parameters, and each size comes in a plain pretrained form and an instruction-following form. A parameter is one learned number inside the model. The design spreads parameters unevenly across the layers of the transformer, the standard neural network for text, so that accuracy improves for a given size. Apple says it wanted to make language model work easier to reproduce and to examine for bias and risk.
How it came about
Apple researchers presented OpenELM in a technical paper first posted on 22 April 2024, with a revised version on 2 May 2024. Alongside the paper, the team put the model weights on Hugging Face and published the full pipeline: data preparation, training, fine-tuning and evaluation, plus training logs and several checkpoints. The models were pretrained with Apple's CoreNet library. Code was also supplied for converting the models to MLX, Apple's library for running and fine-tuning models on its own devices.
Inside the model
The transformer in OpenELM assigns parameters layer by layer rather than giving every layer the same width, which the authors say buys accuracy for the same parameter budget. At roughly one billion parameters they report a 2.36% accuracy gain over OLMo while using half as many pre-training tokens. Pre-training drew on RefinedWeb, a deduplicated version of PILE, and subsets of RedPajama and Dolma v1.6, adding up to about 1.8 trillion tokens. Tokenisation uses the LLaMA tokenizer, which needs a beginning-of-sequence token added. Both raw and instruction-tuned checkpoints are published.
What it can do
Reported zero-shot averages across seven tasks rise from 54.37 for the 270M base model to 69.15 for the 3B instruction-tuned one; the instruction versions generally score above their base counterparts. On the LLM360 and OpenLLM Leaderboard task sets the same pattern holds, with the 3B-Instruct model averaging 51.22 and 55.73 respectively. MMLU scores stay low across the family, around 25 to 27. Results are given as published, without ranking against other models here.
How to use it
All eight checkpoints are downloadable from Hugging Face under the Apple organisation. The accompanying repository covers data preparation, training, fine-tuning and evaluation, and includes training logs and multiple checkpoints. A sample script shows how to generate text, with options such as a repetition penalty, prompt-lookup speculative decoding and a smaller assistant model for faster output. Conversion code targets MLX for use on Apple hardware.
Safety and testing
The authors frame the release partly around openness to risk and bias research, arguing that public weights, data recipes and logs let others probe these issues. The published evaluations include CrowS-Pairs, a benchmark for social bias, alongside standard accuracy tasks. No separate red-teaming or incident reports are described in these pages.
Reception and debate
The pages available here are the project's own model card and paper listing, so they report Apple's claims rather than outside assessment. The authors position the release against prior practice of shipping weights and inference code only, while training on private data. No reviews, lawsuits or bans are described here.
Why it matters
By shipping the whole pipeline rather than just weights, OpenELM is presented as a resource for reproducible language model research. The layer-wise scaling approach is offered as a way to get more accuracy from a fixed parameter budget, illustrated against OLMo at the one-billion-parameter scale. The MLX conversion code ties the models to on-device use on Apple products.
Questions people ask about OpenELM
What is OpenELM?
A family of small language models released by Apple in April 2024. It comes in four sizes, from 270 million to 3 billion parameters, each with a pretrained and an instruction-tuned version. Apple published the weights together with the training and evaluation framework.
What does the name stand for?
The letters expand to Open Efficient Language Models. The 'open' part refers to the release of weights, training recipes, logs and code, not just the final model files.
What data was it trained on?
Roughly 1.8 trillion tokens drawn from RefinedWeb, a deduplicated PILE, a subset of RedPajama and a subset of Dolma v1.6. Apple notes that users should check the licence terms of those datasets themselves.
How do I run it?
Weights are on Hugging Face, and a sample script covers text generation with options like repetition penalty and speculative decoding. There is also code to convert the models to MLX for inference and fine-tuning on Apple devices.
How large is the biggest version?
The largest is OpenELM-3B, with 3 billion parameters. Its instruction-tuned variant averages 69.15 across seven zero-shot tasks as reported in the model card.
Good reading and sources
There is no English Wikipedia article on OpenELM yet, so the facts on this page come from the sources listed above: huggingface.co/apple/OpenELM-3B-Instruct, arxiv.org/abs/2404.14619. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.