News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
InstructGPT

InstructGPT

GPT-3 models fine-tuned with human feedback to follow instructions

InstructGPT is a set of GPT-3 models fine-tuned to follow user instructions, developed by OpenAI and described in a paper posted in March 2022. Human evaluators preferred outputs from the 1.3 billion parameter version over those from the 175 billion parameter GPT-3, showing that alignment training can matter more than raw size.

Share on WhatsApp

  • Fine-tuned GPT-3 with human demonstrations and human rankings of outputs
  • 1.3B InstructGPT preferred over 175B GPT-3 in human evaluations
  • Reported better truthfulness and less toxic output than GPT-3
  • Paper posted to arXiv on 4 March 2022
  • Twenty authors, led by Long Ouyang at OpenAI

What it is

InstructGPT is a family of language models produced by fine-tuning GPT-3 so that it follows what a user actually asks for. OpenAI published the work in March 2022. The models come in several sizes, and the paper highlights a version with 1.3 billion parameters alongside the largest GPT-3 at 175 billion. They are built for general instruction-following across many kinds of prompt. The point of the work is alignment: making a model's outputs match a user's intent rather than simply producing plausible text.

How it came about

The work was carried out at OpenAI by a team led by Long Ouyang, with Jeff Wu, Jan Leike, Paul Christiano, John Schulman and others among the twenty authors. The paper, 'Training language models to follow instructions with human feedback', appeared on arXiv on 4 March 2022. It builds directly on GPT-3 and on earlier reinforcement learning from human feedback work at OpenAI.

Inside the model

The team began with prompts written by human labelers and prompts submitted through the OpenAI API. Labelers then wrote demonstrations of the behaviour wanted from the model, and these were used to fine-tune GPT-3 with supervised learning. Next, the researchers gathered rankings of different model outputs and used those rankings to further train the supervised model with reinforcement learning from human feedback, a technique often shortened to RLHF. The result is a model tuned to be helpful and to reduce unwanted outputs.

What it can do

In human evaluations on the team's prompt distribution, outputs from the 1.3 billion parameter InstructGPT were preferred to those from the 175 billion parameter GPT-3, a model with roughly 100 times more parameters. The paper reports improvements in truthfulness and a drop in toxic output generation, with only small regressions on public NLP datasets. The authors also note that InstructGPT still makes simple errors.

How to use it

The models were made available through the OpenAI API. The paper does not describe downloadable weights or a licence for the weights.

Safety and testing

The paper frames the work as an alignment effort, aimed at outputs that are truthful, non-toxic and helpful. It reports reductions in toxic output generation and improvements in truthfulness for the fine-tuned models, while stating plainly that the models still make simple mistakes.

Reception and debate

The paper's headline result, that a much smaller instruction-tuned model can be preferred over a far larger base model, drew attention to reinforcement learning from human feedback as a practical alignment method. The authors themselves present the approach as promising rather than finished.

Why it matters

InstructGPT established a recipe, supervised fine-tuning followed by reinforcement learning from human feedback, that became a standard way to turn base language models into assistants that follow instructions. The work is closely tied to the development of ChatGPT.

Questions people ask about InstructGPT

What is InstructGPT?

It is a version of GPT-3 that OpenAI fine-tuned so it follows user instructions more reliably. The team used human-written demonstrations and human rankings of model outputs, first with supervised learning and then with reinforcement learning from human feedback.

How is it different from GPT-3?

GPT-3 is a base model trained to predict text. InstructGPT starts from GPT-3 and is further trained on human feedback so its answers track what the user wants. In human evaluations, the smallest InstructGPT was preferred over the largest GPT-3.

How large are the models?

The paper reports several sizes. It highlights a 1.3 billion parameter version and compares against the 175 billion parameter GPT-3, which has about 100 times more parameters.

Can I download InstructGPT?

The work describes access through the OpenAI API. The paper does not describe released weights or a licence for them.

Does it still make mistakes?

Yes. The authors state that InstructGPT still makes simple errors, even though it improves on GPT-3 in truthfulness and reduces toxic outputs.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on InstructGPT yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2203.02155, openai.com/index/instruction-following/. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.