News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
GPT-1

GPT-1

OpenAI's first GPT model, a 117M-parameter decoder-only transformer from 2018.

GPT-1 is OpenAI's first large language model and the origin of the GPT series. Released in June 2018 with the paper "Improving Language Understanding by Generative Pre-Training", it introduced the term generative pre-trained transformer and showed that a two-stage pre-training and fine-tuning approach could beat task-specific supervised models across language tasks.

Share on WhatsApp

  • OpenAI's first large language model, released in June 2018
  • 117 million parameters, 12-layer decoder-only transformer
  • Introduced the term generative pre-trained transformer
  • Scored 72.8 on GLUE against a previous best of 68.9
  • Weights released under the MIT licence

What it is

GPT-1 is the first model in OpenAI's GPT series and the model that gave the generative pre-trained transformer its name. OpenAI published it in June 2018 alongside the paper "Improving Language Understanding by Generative Pre-Training". It is a decoder-only transformer with 117 million parameters, built after Google introduced the transformer design in 2017. The model was made for language understanding work such as judging how two sentences relate, answering questions and spotting paraphrases. Its main contribution was a recipe: learn general language from raw text first, then adapt to a specific task with a small supervised step.

How it came about

Before GPT-1, strong neural language systems relied on supervised learning from large hand-labelled datasets. That made them costly to train and hard to apply where labelled text was scarce, for example in Swahili or Haitian Creole. OpenAI's June 2018 paper proposed a semi-supervised alternative and named it generative pre-training. The model was released that month under the MIT licence. GPT-2 later replaced it in the series.

Inside the model

The architecture is a twelve-layer decoder-only transformer with twelve masked self-attention heads, each with 64-dimensional states, giving 768 dimensions in total. Training used the Adam optimiser rather than plain stochastic gradient descent. The learning rate rose linearly from zero over the first 2,000 updates to a peak of 2.5×10⁻⁴, then fell to zero on a cosine schedule. Training ran in two stages: an unsupervised generative phase that set the initial parameters through a language modelling objective, followed by a supervised discriminative phase that tuned those parameters for a target task. The underlying model stayed task-agnostic, with only minimal changes for each task.

What it can do

GPT-1 beat earlier best results on several language tasks. It gained 5.8% and 1.5% on natural language inference sets such as QNLI and MultiNLI, where a model labels sentence pairs as entailment, contradiction or neutral. It improved by 5.7% on RACE, a question-answering set drawn from school exams, and by 8.9% on the Story Cloze Test. Paraphrase detection on Quora Question Pairs rose 4.2%. On the CoLA acceptability task it scored 45.4 against a previous best of 35.0, and it reached 72.8 on the GLUE multi-task suite, versus 68.9 before.

How to use it

The weights were released under the MIT licence, an open licence with few restrictions. The model came out with the June 2018 research paper.

Why it matters

GPT-1 established the two-stage recipe of generative pre-training followed by supervised fine-tuning that the GPT line is built on. It also introduced the term generative pre-trained transformer. Using a transformer instead of attention-augmented recurrent networks gave the model a more structured memory than recurrent methods allowed, which the authors linked to robust transfer performance across diverse tasks. GPT-2 followed it in the series.

Questions people ask about GPT-1

What does the name GPT-1 mean?

It stands for Generative Pre-trained Transformer 1. The name comes from the June 2018 paper that introduced both the model and the phrase generative pre-trained transformer. It is the first entry in OpenAI's GPT series, and GPT-2 replaced it later.

How is GPT-1 different from the models before it?

Earlier top systems depended on supervised learning from large hand-labelled datasets, which was expensive and weak where labelled text was scarce. GPT-1 used two stages instead: unsupervised pre-training on raw text to set the parameters, then supervised fine-tuning for a chosen task.

How big is GPT-1?

It has 117 million parameters. The architecture is a twelve-layer decoder-only transformer with twelve masked self-attention heads, each with 64-dimensional states, adding up to 768 dimensions.

How well did it perform?

It set new marks on several tasks: 5.8% and 1.5% gains on natural language inference, 5.7% on RACE, 8.9% on the Story Cloze Test, and 4.2% on paraphrase detection. It scored 45.4 on CoLA and 72.8 overall on GLUE.

Can I still get the model?

Yes. OpenAI published the weights under the MIT licence, a permissive open licence, alongside the June 2018 paper.

Good reading and sources

Where this page comes from

The facts on this page come from the Wikipedia article "GPT-1", written by its contributors and used under the CC BY-SA 4.0 licence, and from its Wikidata record (CC0). EveryLLM rewrote the text in its own words, with AI assistance, and compares every page with the article to make sure no passage is copied. It may be reused under the same CC BY-SA 4.0 licence with a link back.

It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

The weekly digest

Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.