News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
Mistral 7B

Mistral 7B

A 7.3 billion parameter open language model from Mistral AI

Mistral 7B is a 7.3 billion parameter language model released by Mistral AI in October 2023 under the Apache 2.0 licence. The company says it beats Llama 2 13B on every benchmark it tested and matches much larger models on reasoning, maths and code.

Share on WhatsApp

  • 7.3 billion parameters, released October 2023 under Apache 2.0
  • Sliding window attention of 4,096 hidden states per layer
  • Grouped-query attention for faster generation
  • Company says it beats Llama 2 13B on every benchmark tested
  • Instruct version fine-tuned only on public HuggingFace datasets

What it is

Mistral 7B is a language model with 7.3 billion parameters, put out by the French company Mistral AI. It is a general text model: you give it text and it produces text. The company positions it as unusually strong for its size, saying it beats Llama 2 13B on every benchmark it ran and often matches Llama 1 34B, a model several times larger. It is also meant to handle code. A second version, Mistral 7B Instruct, was tuned to follow instructions and to chat.

How it came about

Mistral AI announced the model in a blog post and published a paper on arXiv (2310.06825) on 10 October 2023. Eighteen authors are listed on the paper, among them Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch and Guillaume Lample. The company re-ran every Llama evaluation through its own pipeline so the comparison would be like for like. It credits CoreWeave for cluster support, the CINECA/EuroHPC team and the operators of the Leonardo machine for compute, and several open-source and cloud teams for integration help.

Inside the model

The model is a decoder transformer. Two attention tricks cut its cost. Grouped-query attention, or GQA, speeds up generation. Sliding window attention, or SWA, lets each layer look back over the previous 4,096 hidden states, so the compute grows linearly with sequence length rather than quadratically. Stacked layers still reach further back, because a token at one layer sees tokens that themselves saw earlier ones. A fixed window also caps the key-value cache at the window size using rotating buffers, which the company says halves cache memory at a sequence length of 8,192. Changes to FlashAttention and xFormers gave a 2x speed-up at 16k tokens with a 4k window. The instruct version was fine-tuned on public instruction datasets from HuggingFace.

What it can do

Mistral AI reports that the model beats Llama 2 13B on every metric it measured, and Llama 1 34B on many, especially code and reasoning. Its benchmark groups cover commonsense reasoning, world knowledge, reading comprehension, maths and code, plus MMLU, BBH and AGI Eval. On knowledge tests it only ties Llama 2 13B, which the company attributes to its small parameter count limiting how much information it can store. On reasoning, comprehension and MMLU the company says it matches a Llama 2 more than three times its size. Its evaluation differs from the Llama 2 paper on MBPP and TriviaQA.

How to use it

Both models are released under Apache 2.0, which the company describes as usable without restrictions. Weights can be downloaded and run locally with its reference implementation, or served on AWS, GCP or Azure through vLLM and Skypilot. The company also says the base model is easy to fine-tune for any task.

Safety and testing

The company states plainly that the instruct model carries no moderation mechanism. It says it wants to work with the community on guardrails so the model can be deployed where outputs need to be moderated.

Reception and debate

The company's own claims are the main source of praise here: it says the model is the strongest of its size and that its instruct version beats Llama 2 13B Chat on both human and automated judgements.

Why it matters

The release showed that a small open model could hold its own against much larger ones, and the company frames the gain in memory and throughput as equivalent to tripling model size. The Apache 2.0 terms and the fine-tuning demonstration were meant to encourage others to build on it.

Questions people ask about Mistral 7B

Who made Mistral 7B and when did it appear?

Mistral AI, the French company, announced it in October 2023. The blog post and an arXiv paper dated 10 October 2023 describe the model, and eighteen authors are credited on the paper.

How big is it and what is it for?

It has 7.3 billion parameters and handles text. Mistral AI targets general use, including reasoning, maths, reading comprehension and code, and says it holds up against models several times its size.

Can I download and run it myself?

Yes. The weights come under Apache 2.0, which the company calls unrestricted. You can run them locally with its reference implementation or deploy them on AWS, GCP or Azure using vLLM and Skypilot.

What is the difference between the base and instruct versions?

The instruct model was fine-tuned on public instruction datasets from HuggingFace to follow instructions and chat. Mistral AI says it beats Llama 2 13B Chat on human and automated benchmarks, but it ships with no moderation.

Why does it use sliding window attention?

Each layer only attends to the previous 4,096 hidden states, so cost grows linearly with sequence length. Stacked layers still reach further back, and the fixed window lets the cache be capped, saving memory during inference.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on Mistral 7B yet, so the facts on this page come from the sources listed above: mistral.ai/news/announcing-mistral-7b, arxiv.org/abs/2310.06825. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.