News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
Falcon-40B

Falcon-40B

A 40-billion parameter open language model from TII.

Falcon-40B is a causal decoder-only language model with 40 billion parameters, built by the Technology Innovation Institute (TII). Released in 2023, it was trained on one trillion tokens of web data and curated corpora. The model is available under the Apache 2.0 licence, allowing for commercial use without royalties.

Share on WhatsApp

  • Trained on 1,000 billion tokens of data
  • Released under Apache 2.0 licence
  • Uses multiquery attention and FlashAttention
  • Supports English, German, Spanish, and French
  • Requires 85-100GB memory for inference

What it is

Falcon-40B is a large language model containing 40 billion parameters. It was developed by the Technology Innovation Institute in the United Arab Emirates. The model uses a causal decoder-only architecture, meaning it predicts the next token in a sequence. It supports English, German, Spanish, and French primarily, with limited ability in several other European languages. The model is designed for research and as a base for further fine-tuning.

How it came about

Training for the model began in December 2022 and lasted for two months. The developers released the model in 2023 as part of the Falcon series, which also includes smaller and larger variants. The release aimed to provide a high-performance open-source alternative to proprietary models. The team published details of their training methods and datasets to support open science.

Inside the model

The architecture adapts concepts from GPT-3 but introduces specific optimizations. It employs rotary positional embeddings and multiquery attention to improve efficiency. FlashAttention is used to speed up processing. The model was trained on 384 Nvidia A100 GPUs using a custom distributed training codebase called Gigatron. The training data consisted of one trillion tokens, mostly from the RefinedWeb dataset, which is a filtered web crawl.

What it can do

The model performs well on standard language benchmarks, often outperforming other open models of similar or larger size at the time of release. It is capable of text generation, summarization, and coding tasks. However, as a raw pretrained model, it requires fine-tuning to follow instructions effectively. It carries biases present in its web-based training data and may not generalize well to languages outside its primary set.

How to use it

The model weights are publicly available under the Apache 2.0 licence. This permissive licence allows for commercial use without restrictions or royalties. Users can download the model from platforms like Hugging Face. Running inference requires significant memory, estimated at 85 to 100 gigabytes. The developers recommend using PyTorch 2.0 for compatibility.

Safety and testing

The developers note that the model reflects stereotypes and biases found in online data. They advise users to implement guardrails and conduct risk assessments before deploying it in production. The model is not intended for irresponsible or harmful use cases. Fine-tuning is recommended to align the model with specific safety requirements for particular applications.

Reception and debate

The model was noted for its strong performance relative to other open-source options. It ranked highly on community leaderboards such as the OpenLLM Leaderboard. Researchers praised the transparency of the training data and the permissive licensing terms. The release was seen as a significant contribution to the open ecosystem of large language models.

Why it matters

Falcon-40B demonstrated that high-quality web data could rival curated corpora in training effectiveness. Its release encouraged further development of open models with permissive licences. The accompanying RefinedWeb dataset became a widely used resource for training other language models. The architecture choices influenced subsequent designs in the field.

Questions people ask about Falcon-40B

What licence covers Falcon-40B?

The model is released under the Apache 2.0 licence. This permits commercial use, modification, and distribution without requiring royalty payments or imposing strict restrictions on usage.

How much memory is needed to run it?

Running inference with Falcon-40B requires at least 85 to 100 gigabytes of memory. This high requirement is due to the model's size of 40 billion parameters.

What languages does it support?

It primarily supports English, German, Spanish, and French. It has limited capabilities in Italian, Portuguese, Polish, Dutch, Romanian, Czech, and Swedish.

Is it ready for chat applications?

No, this is a raw pretrained model. It should be fine-tuned for specific tasks. For chat applications, the developers recommend using the Falcon-40B-Instruct variant instead.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on Falcon-40B yet, so the facts on this page come from the sources listed above: huggingface.co/tiiuae/falcon-40b, arxiv.org/abs/2311.16867. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.