News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
Falcon Mamba 7B

Falcon Mamba 7B

A 7-billion parameter language model using the Mamba architecture.

The Technology Innovation Institute released Falcon Mamba 7B in October 2024. It is a base language model built on the Mamba architecture rather than the standard Transformer design. The model aims to provide competitive performance with faster inference speeds and lower memory usage for long sequences.

Share on WhatsApp

  • Uses pure Mamba architecture without attention mechanisms
  • Trained on 5.8 trillion tokens from RefinedWeb and other sources
  • Achieves 64.09 average on Open LLM Leaderboard benchmarks
  • Supports unlimited context length at inference time
  • Released under TII Falcon-Mamba License 2.0

What it is

Falcon Mamba 7B is a base large language model developed by the Technology Innovation Institute. It contains seven billion parameters and uses the Mamba architecture, which is a type of state-space model. This design differs from the common Transformer architecture by removing attention mechanisms. The model focuses on English language tasks and was released in October 2024. It is designed to handle long sequences efficiently while maintaining competitive performance against other open-weight models.

How it came about

The Technology Innovation Institute announced the model in October 2024 via a technical report. The development team included Jingwei Zuo, Maksim Velikanov, and others. They aimed to create a pure Mamba-based model that could rival Transformer-based systems. This release followed earlier Falcon models like Falcon 7B and Falcon 11B, extending the family with a new architectural approach. The team made the weights publicly available under a permissive license shortly after the announcement.

Inside the model

The model uses a causal decoder-only design based entirely on the Mamba architecture. It does not use attention mechanisms, relying instead on state-space operations. Training involved 5.8 trillion tokens, primarily from the RefinedWeb dataset. The process used a multi-stage strategy to increase context length from 2,048 to 8,192 tokens. The team employed curriculum learning, selecting data mixtures based on diversity and complexity. Final stages included high-quality technical, code, and math data. Training ran on 256 H100 GPUs for about two months using 3D parallelism.

What it can do

The model performs competitively on standard benchmarks. On the Open LLM Leaderboard, it achieved an average score of 64.09 across tasks like ARC, HellaSwag, and MMLU. It scored 15.04 on a newer set of benchmarks including IFEval and GPQA. The developers claim it surpasses Mistral 7B and Llama 3.1 8B in some metrics. Its architecture allows for faster inference and reduced memory usage during long sequence generation compared to Transformer models. It handles long-range dependencies without a fixed context limit at inference time.

How to use it

The model weights are available for download on Hugging Face. Users can run it on CPUs or GPUs using the Transformers library. The release includes support for various precisions, including FP16 and 4-bit quantization. The license is the TII Falcon-Mamba License 2.0, which is described as permissive. Optimized Mamba kernels are required for best performance, and installation instructions are provided for these components.

Why it matters

Falcon Mamba 7B demonstrates that pure state-space models can compete with Transformer architectures at the 7 billion parameter scale. It challenges the prevailing view that hybrid Mamba-Transformer designs are necessary for top performance. By providing open weights, it enables further research into attention-free language models. The model serves as a reference point for efficiency in long-context processing within the open-source community.

Questions people ask about Falcon Mamba 7B

What makes Falcon Mamba different from other models?

It uses the Mamba architecture, which is a state-space model, instead of the standard Transformer architecture. This means it does not use attention mechanisms, allowing for faster inference and lower memory usage for long sequences.

How many parameters does the model have?

Falcon Mamba 7B has seven billion parameters. It is considered a mid-sized model in the current landscape of large language models, balancing performance with computational efficiency.

Can I use this model commercially?

The model is released under the TII Falcon-Mamba License 2.0, which is described as permissive. You should review the specific license terms on Hugging Face to confirm allowed uses for your specific commercial application.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on Falcon Mamba 7B yet, so the facts on this page come from the sources listed above: huggingface.co/tiiuae/falcon-mamba-7b, arxiv.org/abs/2410.05355. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.