News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
GPT-2

GPT-2

OpenAI's 2019 text model that predicts the next word in a sequence

GPT-2 is a large language model from OpenAI, announced on 14 February 2019 as the second entry in its GPT series. Trained on 8 million web pages, it showed that scaling up parameters and data gave one model many abilities at once, and its staged release became a landmark debate about open weights and misuse.

Share on WhatsApp

  • Announced 14 February 2019; full 1.5B model released 5 November 2019
  • Trained on WebText, built from Reddit-linked pages with at least 3 karma
  • Four sizes: 124M, 355M, 774M and 1.5 billion parameters
  • Weights published under the MIT licence after a staged release
  • Reported training cost of about $43,000 on 32 TPU v3 chips

What it is

Generative Pre-trained Transformer 2, usually shortened to GPT-2, is a large language model built by OpenAI and the second model in its foundational GPT line. It was announced on 14 February 2019 and trained on a corpus of 8 million web pages. The largest version holds 1.5 billion parameters, and smaller releases of 774 million, 355 million and 124 million parameters also exist. GPT-2 is a general-purpose text model: it learned only to predict the next item in a sequence, and that single skill let it translate, answer questions, summarise and write passages of prose.

How it came about

GPT-2 was a direct scale-up of GPT-1, with roughly ten times the parameters and ten times the training data. OpenAI announced it on 14 February 2019 but withheld the code and weights at first, giving selected press only an interface. A 774-million-parameter version followed on 20 August 2019, and the full 1.5-billion-parameter model arrived on 5 November 2019 after OpenAI reported no strong signs of misuse. Smaller 124M and 355M versions were also published. On 9 May 2023 OpenAI released a mapped version in which GPT-4 had been used to identify the role of each neuron.

Inside the model

GPT-2 uses a generative pre-trained transformer design, a deep neural network built on attention rather than the recurrence and convolutions of earlier systems. Attention lets the model weigh the parts of the input most relevant to what it is predicting, which allows heavy parallel training. Its corpus, WebText, was built from pages linked from Reddit posts with at least 3 karma before December 2017, then cleaned of HTML, duplicates and Wikipedia pages. Andrej Karpathy reported training on 32 TPU v3 chips for 168 hours, around $43,000 at roughly $8 per chip-hour.

What it can do

Because it learned only next-item prediction, GPT-2 handled translation, question answering, summarisation and free-form writing without task-specific training. A University of Amsterdam study using a modified Turing test found participants sometimes could not tell its poems from human ones. On WMT-14 it scored 5 BLEU English-to-French and 11.5 BLEU French-to-English, despite only about 10MB of French in its data. Long outputs were its weak point: prose drifted, repeated itself and lost coherence over several paragraphs, and it answered questions less well than retrieval-based systems.

How to use it

GPT-2 is published under the MIT licence, with code on GitHub and weights downloadable from Hugging Face. OpenAI released it in stages rather than all at once: a limited interface first, then the 774M model in August 2019, then the full 1.5B model in November 2019. The small 124M and medium 355M versions can also be downloaded. Later GPT models are not open weights and are reached through APIs instead.

Safety and testing

OpenAI withheld the model at announcement over fears of misuse, such as spam that evades filters, fake reviews and obscene or racist text. It showed a fine-tuned version producing endless positive or negative product reviews. Critics, including Anima Anandkumar, said the danger was overstated and called the withholding the opposite of open; The Gradient published an open letter asking for release. The Allen Institute for Artificial Intelligence responded with a detector for neural fake news. By November 2019 OpenAI said it had seen no strong evidence of misuse.

Reception and debate

Reviewers found the writing striking but flawed. The Verge called the output usually identifiable as non-human yet one of the most exciting examples of language generation, praising its flexibility at translation, summarisation and trivia. The Guardian described plausible newspaper prose, and Vox called it both impressive and a threat to jobs. The Register felt a reader would soon notice something was off. Debate over the partial release was sharp, with some arguing the risks were exaggerated and others warning of floods of plausible prose.

Why it matters

GPT-2 was widely reused before its full release. The r/SubsimulatorGPT2 forum had model instances posting and replying to each other, and a 2019 code autocomplete tool built on it was well received. AI Dungeon launched in 2019 on GPT-2 to generate text adventures, and its company Latitude raised $3.3 million in seed funding in 2021. A replication, OpenGPT-2, appeared with the OpenWebText corpus. Hugging Face distilled it into DistilGPT2, 33% smaller and twice as fast. A teen crisis centre used a GPT-2 chatbot to train counsellors.

Questions people ask about GPT-2

What is GPT-2?

It is a large language model from OpenAI, announced in February 2019 as the second in its GPT series. Trained on 8 million web pages, it predicts the next item in a sequence, which gives it abilities such as translation, summarisation, question answering and writing prose.

How big is GPT-2?

The largest version has 1.5 billion parameters and was trained on 40 gigabytes of text. Three smaller versions were also released, at 774 million, 355 million and 124 million parameters. The full model is over five gigabytes in size and needs substantial memory to run.

Can I download it?

Yes. After a staged release, OpenAI published the weights under the MIT licence, and the versions are downloadable from Hugging Face, with code on GitHub. This differs from later GPT models, whose weights were never made public and which are offered through APIs.

Why did OpenAI hold it back at first?

It cited the risk of malicious use, such as spam that slips past filters, fake product reviews and obscene or racist text. Some researchers argued the threat was exaggerated. By November 2019 OpenAI said it had seen no strong evidence of misuse and released the full model.

What are its weaknesses?

Long passages tend to repeat, stray off topic and lose coherence after a couple of paragraphs. It also answers questions less well than systems built to extract and retrieve information. Running it is heavy work: a single prediction can pin a CPU for minutes, and even on a GPU it takes seconds.

Good reading and sources

Where this page comes from

The facts on this page come from the Wikipedia article "GPT-2", written by its contributors and used under the CC BY-SA 4.0 licence, and from its Wikidata record (CC0). EveryLLM rewrote the text in its own words, with AI assistance, and compares every page with the article to make sure no passage is copied. It may be reused under the same CC BY-SA 4.0 licence with a link back.

It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

The weekly digest

Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.