News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
OPT

OPT

Meta AI's family of open decoder-only language models from 125M to 175B parameters

OPT (Open Pre-trained Transformers) is a family of decoder-only pre-trained language models released by Meta AI in May 2022, ranging from 125 million to 175 billion parameters. The team says OPT-175B matches GPT-3 performance while producing about one seventh of the carbon footprint, and the weights were shared with researchers to enable reproducible study of large language models.

Share on WhatsApp

  • Family of decoder-only models from 125M to 175B parameters
  • OPT-175B reported as comparable to GPT-3
  • About 1/7th the carbon footprint of developing GPT-3, the team says
  • Trained on 180B tokens (800GB) of filtered text
  • OPT-175B trained on 992 80GB A100 GPUs for about 33 days

What it is

OPT stands for Open Pre-trained Transformers. It is a suite of decoder-only language models built by Meta AI and first released in May 2022. The family spans sizes from 125 million up to 175 billion parameters. The largest member, OPT-175B, was designed to sit in the same performance and size class as GPT-3. The models are pre-trained for text generation and can also be prompted for zero-shot and few-shot evaluation of downstream tasks, or fine-tuned on a specific task. The team says the goal was to let more researchers study large language models directly rather than only through paid APIs.

How it came about

The OPT work was led by Susan Zhang with 18 co-authors at Meta AI, and the paper was submitted to arXiv on 2 May 2022, with a later revision in June 2022. The models were first released in the metaseq repository on 3 May 2022. The team published an official model card as an appendix to the paper, along with a logbook describing the infrastructure problems they hit during training and code for experimenting with the released checkpoints.

Inside the model

OPT models are decoder-only transformers trained with a causal language modelling objective, the same self-supervised setup used for GPT-3. Text is tokenised with the GPT-2 byte-level version of byte pair encoding, with a vocabulary of 50,272, and inputs are sequences of 2,048 consecutive tokens. Training data is a union of five filtered text sets: BookCorpus, CC-Stories, parts of The Pile, the Pushshift.io Reddit dataset and CCNewsV2. The final corpus holds 180B tokens, about 800GB. OPT-175B was trained on 992 80GB A100 GPUs for roughly 33 days.

What it can do

The team reports that OPT-175B is comparable to GPT-3, using the same prompts and experimental setup for evaluation. The models support zero-shot and few-shot prompting as well as text generation. The model card notes real limits: OPT-175B can show bias, weak generation diversity and hallucination, and it inherits the problems common to large language models trained on unfiltered internet text. Demonstrations in the model card show gendered differences in job completions, and the team states this bias carries over into fine-tuned versions.

How to use it

OPT was released with the aim of fully and responsibly sharing the weights with interested researchers. Checkpoints from 125M to 175B are hosted on the Hugging Face model hub, and code for running them is available in the metaseq repository. The official model card appears in Appendix D of the paper. For the larger checkpoints, half-precision loading on GPU is recommended rather than the standard text-generation pipeline.

Safety and testing

The team frames the release itself as a safety measure, arguing that definitions of risk, harm, bias and toxicity need to be set by the research community, which requires models to be available for study. The model card warns that the training corpus includes unfiltered Common Crawl and Reddit content that may be insulting or threatening, and that OPT-175B carries bias, hallucination and generation diversity problems. The release also included a logbook of infrastructure challenges.

Reception and debate

The release was presented as a reproducibility effort, giving researchers full weights rather than API access. The team's own model card is candid about weaknesses, describing bias and safety limitations in OPT-175B and noting that the model is not immune to the issues affecting modern large language models. No formal reception details are recorded here.

Why it matters

OPT was positioned as a way to bring more voices into the study of large language models, since restricted access had limited work on robustness, bias and toxicity. By matching the GPT-3 class at a fraction of the reported carbon footprint, and by publishing weights, code and a training logbook, it offered a reproducible reference point for researchers studying how and why these models behave as they do.

Questions people ask about OPT

What is OPT?

OPT stands for Open Pre-trained Transformers. It is a family of decoder-only language models released by Meta AI in May 2022, with sizes from 125 million to 175 billion parameters. The largest, OPT-175B, was built to match the GPT-3 class of models.

Who can use OPT?

The weights were shared with interested researchers, and checkpoints are hosted on the Hugging Face model hub with code in the metaseq repository. The team says the aim was fully and responsible sharing, and an official model card is included as an appendix to the paper.

What data was OPT trained on?

The corpus is a union of five filtered text sets: BookCorpus, CC-Stories, parts of The Pile, the Pushshift.io Reddit dataset and CCNewsV2. The final training data holds 180B tokens, about 800GB, with a 200MB validation split sampled in proportion to each dataset.

What are OPT's main limitations?

The model card states that OPT-175B has bias and safety limitations, can hallucinate, and can produce weak generation diversity. Because the training data includes unfiltered Common Crawl and Reddit content, biased outputs appear, and the team says this bias also affects fine-tuned versions.

How does OPT compare to GPT-3?

The team reports OPT-175B as comparable to GPT-3 in performance, using the same prompts and experimental setup for evaluation. They also say it required only about one seventh of the carbon footprint to develop.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on OPT yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2205.01068, huggingface.co/facebook/opt-66b. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.