News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
Falcon 180B

Falcon 180B

Open 180 billion parameter language model from TII

Falcon 180B is a large causal decoder-only language model with 180 billion parameters. Released in late 2023 by the Technology Innovation Institute, it was trained on over 3.5 trillion tokens. The model aims to provide high performance for open science under a permissive licence.

Share on WhatsApp

  • Trained on over 3.5 trillion tokens of text
  • Uses up to 4,096 A100 GPUs on AWS cloud
  • Released under a permissive open licence
  • 180 billion parameters in a decoder-only design
  • Outperforms PaLM and Chinchilla in evaluations

What it is

Falcon 180B is a general-purpose language model with 180 billion parameters. It belongs to the Falcon series developed by the Technology Innovation Institute. The model uses a causal decoder-only architecture. It was trained on a diverse set of high-quality data, mostly from the web. This dataset totals more than 3.5 trillion tokens. The release aims to support open science and accelerate the growth of an open ecosystem for large language models.

How it came about

The model was introduced in November 2023 as part of the Falcon series. This family includes smaller versions with 7 billion and 40 billion parameters. Falcon 180B represents the largest openly documented pretraining run at the time of its release. The developers released the model alongside a 600 billion token extract of their web dataset. This move was designed to foster transparency and allow other researchers to build upon their work.

Inside the model

The architecture is a causal decoder-only transformer. Training relied on custom distributed code that allowed efficient use of cloud infrastructure. The team used up to 4,096 Nvidia A100 GPUs on Amazon Web Services. This setup handled limited interconnect bandwidth effectively. The training corpus was assembled predominantly from web data. The developers focused on creating a high-quality dataset to improve model performance while managing costs.

What it can do

The model significantly outperforms earlier systems like PaLM and Chinchilla. It also improves upon concurrently developed models such as LLaMA 2 and Inflection-1. Its performance nears that of PaLM-2-Large but at a reduced cost for pretraining and inference. The developers claim it is one of the three best language models globally, alongside GPT-4 and PaLM-2-Large. Detailed evaluations were reported to support these claims.

How to use it

The model weights are available under a permissive licence. This allows for broad use in research and commercial applications. The developers also released a subset of their training data. This 600 billion token extract helps others understand the data foundation. The goal is to lower barriers for entry into large model development and encourage community collaboration.

Why it matters

Falcon 180B demonstrated that large-scale models could be trained efficiently on cloud infrastructure with limited interconnect. By releasing the weights and data openly, it contributed to the open-source AI community. It set a benchmark for openly documented pretraining runs. The model provided a strong alternative to proprietary systems for organisations seeking high performance without closed access.

Questions people ask about Falcon 180B

How many parameters does Falcon 180B have?

The model contains 180 billion parameters. It is the largest member of the Falcon series, which also includes 7 billion and 40 billion parameter versions.

What data was used to train the model?

It was trained on over 3.5 trillion tokens. The data consists of diverse high-quality corpora assembled predominantly from web sources. A 600 billion token extract was released publicly.

Is Falcon 180B open source?

The model weights are released under a permissive licence. This allows for open use and modification. The developers also shared part of the training dataset to support open science.

How does it compare to other models?

It outperforms PaLM and Chinchilla. It also improves on LLaMA 2 and Inflection-1. Its performance is close to PaLM-2-Large but with lower training and inference costs.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on Falcon 180B yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2311.16867. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.