Llemma
Open language model for mathematical reasoning
EleutherAI released Llemma in October 2023 as a specialised language model for mathematics. Built by continuing the pretraining of Code Llama on mathematical data, it offers open weights in 7 billion and 34 billion parameter sizes. The model excels at chain-of-thought reasoning and formal theorem proving.
- Trained on 50 billion tokens of mathematical data
- Outperforms Code Llama on math benchmarks
- Supports formal theorem proving without fine-tuning
- Available in 7B and 34B parameter sizes
- All artifacts including data are openly released
What it is
Llemma is a large language model designed specifically for mathematical tasks. EleutherAI developed the system by taking existing Code Llama weights and training them further on a specialised dataset called Proof-Pile-2. This process created two versions with 7 billion and 34 billion parameters. The model focuses on solving complex mathematical problems through logical reasoning steps rather than just predicting the next word.
How it came about
The project was announced in October 2023 via a research paper submitted to arXiv. A team of researchers including Zhangir Azerbayev and Sean Welleck led the development. They built upon the Code Llama architecture, which itself is derived from earlier foundational models. The team released all artifacts openly, including the training data and experimental code, to allow other researchers to replicate their results.
Inside the model
The architecture uses a decoder-only transformer design inherited from Code Llama. Training involved processing 50 billion tokens from the Proof-Pile-2 dataset. This data mixture includes scientific papers, web content containing mathematics, and mathematical code. The model was not fine-tuned for specific tools but learned to use computational aids like Python and formal theorem provers during its pretraining phase.
What it can do
Llemma performs strongly on chain-of-thought mathematics benchmarks. On the GSM8k test, the 34 billion parameter version scored 51.5 per cent, outperforming similarly sized Code Llama. It also achieved 25.0 per cent on the MATH benchmark. The model can use external tools for calculations and engage in formal theorem proving without additional instruction tuning. Majority voting techniques further improved its accuracy on these tasks.
How to use it
EleutherAI provides open access to the model weights for both the 7 billion and 34 billion parameter variants. The organisation also released the Proof-Pile-2 dataset and the code used for experiments. Users can download these assets from platforms like Hugging Face to run the model locally or integrate it into their own research workflows.
Why it matters
Llemma demonstrated that open base models could compete with larger, unreleased systems in mathematical reasoning. By releasing the training data and code, the project supported reproducibility in academic research. It set a new standard for open-source performance on equi-parameter benchmarks against proprietary models like Minerva.
Questions people ask about Llemma
What data was used to train Llemma?
The model was trained on the Proof-Pile-2 dataset. This collection includes scientific papers, web pages with mathematical content, and mathematical code. The training process used 50 billion tokens from this mixture.
How does Llemma compare to other models?
On chain-of-thought math tasks, Llemma outperforms Llama-2 and Code Llama. When controlled for model size, it also beats the Minerva model suite on several benchmarks such as GSM8k and MATH.
Can Llemma use external tools?
Yes, the model has strong capabilities in using computational tools. It can write Python code and interact with formal theorem provers to solve mathematical problems without needing any further fine-tuning.
Good reading and sources
There is no English Wikipedia article on Llemma yet, so the facts on this page come from the sources listed above: huggingface.co/EleutherAI/llemma_34b, arxiv.org/abs/2310.10631. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.