Falcon 180B
Open 180 billion parameter language model from TII
Falcon 180B is a large causal decoder-only language model with 180 billion parameters. Released in late 2023 by the Technology Innovation Institute, it was trained on over 3.5 trillion tokens. The model aims to provide high performance for open science under a permissive licence.
- Trained on over 3.5 trillion tokens of text
- Uses up to 4,096 A100 GPUs on AWS cloud
- Released under a permissive open licence
- 180 billion parameters in a decoder-only design
- Outperforms PaLM and Chinchilla in evaluations
What it is
Falcon 180B is a general-purpose language model with 180 billion parameters. It belongs to the Falcon series developed by the Technology Innovation Institute. The model uses a causal decoder-only architecture. It was trained on a diverse set of high-quality data, mostly from the web. This dataset totals more than 3.5 trillion tokens. The release aims to support open science and accelerate the growth of an open ecosystem for large language models.
How it came about
The model was introduced in November 2023 as part of the Falcon series. This family includes smaller versions with 7 billion and 40 billion parameters. Falcon 180B represents the largest openly documented pretraining run at the time of its release. The developers released the model alongside a 600 billion token extract of their web dataset. This move was designed to foster transparency and allow other researchers to build upon their work.
Inside the model
The architecture is a causal decoder-only transformer. Training relied on custom distributed code that allowed efficient use of cloud infrastructure. The team used up to 4,096 Nvidia A100 GPUs on Amazon Web Services. This setup handled limited interconnect bandwidth effectively. The training corpus was assembled predominantly from web data. The developers focused on creating a high-quality dataset to improve model performance while managing costs.
What it can do
The model significantly outperforms earlier systems like PaLM and Chinchilla. It also improves upon concurrently developed models such as LLaMA 2 and Inflection-1. Its performance nears that of PaLM-2-Large but at a reduced cost for pretraining and inference. The developers claim it is one of the three best language models globally, alongside GPT-4 and PaLM-2-Large. Detailed evaluations were reported to support these claims.
How to use it
The model weights are available under a permissive licence. This allows for broad use in research and commercial applications. The developers also released a subset of their training data. This 600 billion token extract helps others understand the data foundation. The goal is to lower barriers for entry into large model development and encourage community collaboration.
Why it matters
Falcon 180B demonstrated that large-scale models could be trained efficiently on cloud infrastructure with limited interconnect. By releasing the weights and data openly, it contributed to the open-source AI community. It set a benchmark for openly documented pretraining runs. The model provided a strong alternative to proprietary systems for organisations seeking high performance without closed access.
Questions people ask about Falcon 180B
How many parameters does Falcon 180B have?
The model contains 180 billion parameters. It is the largest member of the Falcon series, which also includes 7 billion and 40 billion parameter versions.
What data was used to train the model?
It was trained on over 3.5 trillion tokens. The data consists of diverse high-quality corpora assembled predominantly from web sources. A 600 billion token extract was released publicly.
Is Falcon 180B open source?
The model weights are released under a permissive licence. This allows for open use and modification. The developers also shared part of the training dataset to support open science.
How does it compare to other models?
It outperforms PaLM and Chinchilla. It also improves on LLaMA 2 and Inflection-1. Its performance is close to PaLM-2-Large but with lower training and inference costs.
Good reading and sources
There is no English Wikipedia article on Falcon 180B yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2311.16867. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.