Falcon-40B
A 40-billion parameter open language model from TII.
Falcon-40B is a causal decoder-only language model with 40 billion parameters, built by the Technology Innovation Institute (TII). Released in 2023, it was trained on one trillion tokens of web data and curated corpora. The model is available under the Apache 2.0 licence, allowing for commercial use without royalties.
- Trained on 1,000 billion tokens of data
- Released under Apache 2.0 licence
- Uses multiquery attention and FlashAttention
- Supports English, German, Spanish, and French
- Requires 85-100GB memory for inference
What it is
Falcon-40B is a large language model containing 40 billion parameters. It was developed by the Technology Innovation Institute in the United Arab Emirates. The model uses a causal decoder-only architecture, meaning it predicts the next token in a sequence. It supports English, German, Spanish, and French primarily, with limited ability in several other European languages. The model is designed for research and as a base for further fine-tuning.
How it came about
Training for the model began in December 2022 and lasted for two months. The developers released the model in 2023 as part of the Falcon series, which also includes smaller and larger variants. The release aimed to provide a high-performance open-source alternative to proprietary models. The team published details of their training methods and datasets to support open science.
Inside the model
The architecture adapts concepts from GPT-3 but introduces specific optimizations. It employs rotary positional embeddings and multiquery attention to improve efficiency. FlashAttention is used to speed up processing. The model was trained on 384 Nvidia A100 GPUs using a custom distributed training codebase called Gigatron. The training data consisted of one trillion tokens, mostly from the RefinedWeb dataset, which is a filtered web crawl.
What it can do
The model performs well on standard language benchmarks, often outperforming other open models of similar or larger size at the time of release. It is capable of text generation, summarization, and coding tasks. However, as a raw pretrained model, it requires fine-tuning to follow instructions effectively. It carries biases present in its web-based training data and may not generalize well to languages outside its primary set.
How to use it
The model weights are publicly available under the Apache 2.0 licence. This permissive licence allows for commercial use without restrictions or royalties. Users can download the model from platforms like Hugging Face. Running inference requires significant memory, estimated at 85 to 100 gigabytes. The developers recommend using PyTorch 2.0 for compatibility.
Safety and testing
The developers note that the model reflects stereotypes and biases found in online data. They advise users to implement guardrails and conduct risk assessments before deploying it in production. The model is not intended for irresponsible or harmful use cases. Fine-tuning is recommended to align the model with specific safety requirements for particular applications.
Reception and debate
The model was noted for its strong performance relative to other open-source options. It ranked highly on community leaderboards such as the OpenLLM Leaderboard. Researchers praised the transparency of the training data and the permissive licensing terms. The release was seen as a significant contribution to the open ecosystem of large language models.
Why it matters
Falcon-40B demonstrated that high-quality web data could rival curated corpora in training effectiveness. Its release encouraged further development of open models with permissive licences. The accompanying RefinedWeb dataset became a widely used resource for training other language models. The architecture choices influenced subsequent designs in the field.
Questions people ask about Falcon-40B
What licence covers Falcon-40B?
The model is released under the Apache 2.0 licence. This permits commercial use, modification, and distribution without requiring royalty payments or imposing strict restrictions on usage.
How much memory is needed to run it?
Running inference with Falcon-40B requires at least 85 to 100 gigabytes of memory. This high requirement is due to the model's size of 40 billion parameters.
What languages does it support?
It primarily supports English, German, Spanish, and French. It has limited capabilities in Italian, Portuguese, Polish, Dutch, Romanian, Czech, and Swedish.
Is it ready for chat applications?
No, this is a raw pretrained model. It should be fine-tuned for specific tasks. For chat applications, the developers recommend using the Falcon-40B-Instruct variant instead.
Good reading and sources
There is no English Wikipedia article on Falcon-40B yet, so the facts on this page come from the sources listed above: huggingface.co/tiiuae/falcon-40b, arxiv.org/abs/2311.16867. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.