News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
Mixtral 8x7B

Mixtral 8x7B

Sparse mixture-of-experts language model with open weights

Mixtral 8x7B is a sparse mixture-of-experts language model released by Mistral AI under an Apache 2.0 licence. It uses 46.7B total parameters but only 12.9B per token, and Mistral AI says it matches or beats Llama 2 70B and GPT-3.5 on most benchmarks.

Share on WhatsApp

  • 46.7B total parameters, 12.9B active per token
  • 8 experts per layer, 2 chosen by a router for each token
  • 32k-token context window
  • Apache 2.0 licence for base and instruct models
  • 8.30 on MT-Bench for the instruction-tuned version

What it is

Mixtral 8x7B is a language model built by Mistral AI and released with open weights under the Apache 2.0 licence. It is a sparse mixture of experts: the network holds 46.7B parameters in total, but only about 12.9B of them are used for any single token. Mistral AI positions it as a general model that handles a 32k-token context, works in English, French, Italian, German and Spanish, and is strong at writing code. A second version, Mixtral 8x7B Instruct, was tuned to follow instructions.

How it came about

Mistral AI released Mixtral 8x7B as part of its effort to put original model designs in the hands of developers rather than reusing familiar training recipes. The company published a technical paper, Mixtral of Experts, on arXiv in January 2024, written by a team of 26 authors led by Albert Q. Jiang. Both the base model and the instruction-tuned variant were published together under Apache 2.0. Mistral AI credits CoreWeave and Scaleway for technical support during training.

Inside the model

Mixtral keeps the architecture of Mistral 7B but replaces each feedforward block with eight separate blocks, called experts. For every token at every layer, a small router network picks two of the eight and adds their outputs together. The chosen pair can change from one token to the next. This raises total capacity while keeping speed and cost close to a 12.9B model, since most parameters sit idle. Experts and routers are trained at the same time on text taken from the open web. The instruct version was refined with supervised fine-tuning and direct preference optimisation.

What it can do

Mistral AI reports that Mixtral matches or exceeds Llama 2 70B and GPT-3.5 across the benchmarks it evaluated, with the widest gaps in mathematics, code generation and multilingual tasks. The instruction-tuned model scored 8.30 on MT-Bench, which the company describes as the best open-source result at the time, close to GPT-3.5. On BBQ, a bias benchmark, the base model showed less bias than Llama 2, and on BOLD it produced more positive sentiment with similar variance.

How to use it

Both models are downloadable under Apache 2.0. Mistral AI submitted changes to the vLLM project, adding Megablocks CUDA kernels for efficient inference, and SkyPilot can deploy vLLM endpoints in the cloud. The company also ran Mixtral 8x7B behind its mistral-small endpoint, offered in beta.

Safety and testing

Mistral AI measured the base model on BBQ and BOLD to surface bias and sentiment issues that fine-tuning or preference modelling could address. The company notes that prompting can block unwanted outputs in applications needing strict moderation, and that preference tuning serves the same purpose, but warns that without such a prompt the model follows whatever instructions it is given.

Why it matters

Mixtral 8x7B was presented as the strongest open-weight model available under a permissive licence at the time of release, and as evidence that sparse expert designs can raise quality without raising inference cost. Mistral AI placed it in a family with Mistral 7B, describing both as unusually efficient compared with the Llama 2 line.

Questions people ask about Mixtral 8x7B

What does 'sparse mixture of experts' mean here?

Each layer holds eight separate feedforward blocks, called experts. A router picks two of them for every token and adds their outputs. Most parameters stay unused for any given token, so the model runs at roughly the speed and cost of a 12.9B model despite holding 46.7B parameters.

How does it compare with Llama 2 70B and GPT-3.5?

Mistral AI says Mixtral matches or outperforms both on most benchmarks it tested, with the largest advantages in maths, code generation and multilingual tasks. It also claims six times faster inference than Llama 2 70B. These are the company's own reported results.

What licence are the weights released under?

Apache 2.0, for both the base model and the instruction-tuned Mixtral 8x7B Instruct. That is a permissive open-source licence allowing broad reuse.

Which languages does it support?

English, French, Italian, German and Spanish. Mistral AI highlights strong performance on multilingual benchmarks among these.

How do I run it?

Weights can be downloaded and served with vLLM, which Mistral AI extended with Megablocks CUDA kernels for efficient inference, and deployed via SkyPilot on cloud instances. Mistral AI also offered it through the mistral-small endpoint in beta.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on Mixtral 8x7B yet, so the facts on this page come from the sources listed above: mistral.ai/news/mixtral-of-experts, arxiv.org/abs/2401.04088. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.