OPT
Meta AI's family of open decoder-only language models from 125M to 175B parameters
OPT (Open Pre-trained Transformers) is a family of decoder-only pre-trained language models released by Meta AI in May 2022, ranging from 125 million to 175 billion parameters. The team says OPT-175B matches GPT-3 performance while producing about one seventh of the carbon footprint, and the weights were shared with researchers to enable reproducible study of large language models.
- Family of decoder-only models from 125M to 175B parameters
- OPT-175B reported as comparable to GPT-3
- About 1/7th the carbon footprint of developing GPT-3, the team says
- Trained on 180B tokens (800GB) of filtered text
- OPT-175B trained on 992 80GB A100 GPUs for about 33 days
What it is
OPT stands for Open Pre-trained Transformers. It is a suite of decoder-only language models built by Meta AI and first released in May 2022. The family spans sizes from 125 million up to 175 billion parameters. The largest member, OPT-175B, was designed to sit in the same performance and size class as GPT-3. The models are pre-trained for text generation and can also be prompted for zero-shot and few-shot evaluation of downstream tasks, or fine-tuned on a specific task. The team says the goal was to let more researchers study large language models directly rather than only through paid APIs.
How it came about
The OPT work was led by Susan Zhang with 18 co-authors at Meta AI, and the paper was submitted to arXiv on 2 May 2022, with a later revision in June 2022. The models were first released in the metaseq repository on 3 May 2022. The team published an official model card as an appendix to the paper, along with a logbook describing the infrastructure problems they hit during training and code for experimenting with the released checkpoints.
Inside the model
OPT models are decoder-only transformers trained with a causal language modelling objective, the same self-supervised setup used for GPT-3. Text is tokenised with the GPT-2 byte-level version of byte pair encoding, with a vocabulary of 50,272, and inputs are sequences of 2,048 consecutive tokens. Training data is a union of five filtered text sets: BookCorpus, CC-Stories, parts of The Pile, the Pushshift.io Reddit dataset and CCNewsV2. The final corpus holds 180B tokens, about 800GB. OPT-175B was trained on 992 80GB A100 GPUs for roughly 33 days.
What it can do
The team reports that OPT-175B is comparable to GPT-3, using the same prompts and experimental setup for evaluation. The models support zero-shot and few-shot prompting as well as text generation. The model card notes real limits: OPT-175B can show bias, weak generation diversity and hallucination, and it inherits the problems common to large language models trained on unfiltered internet text. Demonstrations in the model card show gendered differences in job completions, and the team states this bias carries over into fine-tuned versions.
How to use it
OPT was released with the aim of fully and responsibly sharing the weights with interested researchers. Checkpoints from 125M to 175B are hosted on the Hugging Face model hub, and code for running them is available in the metaseq repository. The official model card appears in Appendix D of the paper. For the larger checkpoints, half-precision loading on GPU is recommended rather than the standard text-generation pipeline.
Safety and testing
The team frames the release itself as a safety measure, arguing that definitions of risk, harm, bias and toxicity need to be set by the research community, which requires models to be available for study. The model card warns that the training corpus includes unfiltered Common Crawl and Reddit content that may be insulting or threatening, and that OPT-175B carries bias, hallucination and generation diversity problems. The release also included a logbook of infrastructure challenges.
Reception and debate
The release was presented as a reproducibility effort, giving researchers full weights rather than API access. The team's own model card is candid about weaknesses, describing bias and safety limitations in OPT-175B and noting that the model is not immune to the issues affecting modern large language models. No formal reception details are recorded here.
Why it matters
OPT was positioned as a way to bring more voices into the study of large language models, since restricted access had limited work on robustness, bias and toxicity. By matching the GPT-3 class at a fraction of the reported carbon footprint, and by publishing weights, code and a training logbook, it offered a reproducible reference point for researchers studying how and why these models behave as they do.
Questions people ask about OPT
What is OPT?
OPT stands for Open Pre-trained Transformers. It is a family of decoder-only language models released by Meta AI in May 2022, with sizes from 125 million to 175 billion parameters. The largest, OPT-175B, was built to match the GPT-3 class of models.
Who can use OPT?
The weights were shared with interested researchers, and checkpoints are hosted on the Hugging Face model hub with code in the metaseq repository. The team says the aim was fully and responsible sharing, and an official model card is included as an appendix to the paper.
What data was OPT trained on?
The corpus is a union of five filtered text sets: BookCorpus, CC-Stories, parts of The Pile, the Pushshift.io Reddit dataset and CCNewsV2. The final training data holds 180B tokens, about 800GB, with a 200MB validation split sampled in proportion to each dataset.
What are OPT's main limitations?
The model card states that OPT-175B has bias and safety limitations, can hallucinate, and can produce weak generation diversity. Because the training data includes unfiltered Common Crawl and Reddit content, biased outputs appear, and the team says this bias also affects fine-tuned versions.
How does OPT compare to GPT-3?
The team reports OPT-175B as comparable to GPT-3 in performance, using the same prompts and experimental setup for evaluation. They also say it required only about one seventh of the carbon footprint to develop.
Good reading and sources
There is no English Wikipedia article on OPT yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2205.01068, huggingface.co/facebook/opt-66b. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.