News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
Phi-2

Phi-2

A 2.7 billion parameter Microsoft research model for QA, chat and code

Phi-2 is a compact text-generating Transformer built by Microsoft, with 2.7 billion parameters. Released as an open research model, it targets common sense, language understanding and reasoning tasks, and was put out to let researchers study safety problems such as toxicity and bias in small models.

Share on WhatsApp

  • 2.7 billion parameters, released openly by Microsoft
  • Trained on 250 billion tokens of synthetic text and web data
  • No RLHF and no instruction tuning applied
  • MIT licence, supported in transformers 4.37.0
  • Known FP16 attention overflow issue

What it is

Phi-2 is a language model with 2.7 billion parameters, built on the Transformer design and trained to predict the next word. Microsoft released it openly for research use. It is aimed at everyday reasoning, reading comprehension and code writing, and the company positions it as a strong performer among models below 13 billion parameters on tests of common sense, language understanding and logic. Because it is small, it is cheap to run and easy to study. It is a base model, so it has not been tuned to follow instructions or to chat politely.

How it came about

Phi-2 continues the Phi line from Microsoft, following Phi-1.5. It reused that model's training sources and added a further collection of synthetic language data plus web pages chosen for safety and educational worth. The team published the weights openly so that researchers could examine safety questions on a small, unrestricted model. Microsoft says the goal was to support work on lowering toxicity, understanding social bias and improving controllability.

Inside the model

The system is a Transformer that learns by guessing the next word in a sequence. Its training set totals 250 billion tokens. It combines synthetic text produced by AOAI GPT-3.5 with web corpora drawn from Falcon RefinedWeb and SlimPajama, and those web pages were reviewed by AOAI GPT-4. Training used PyTorch, DeepSpeed and Flash-Attention. No reinforcement learning from human feedback was applied, and there was no instruction tuning, so the raw pretrained behaviour is what users get.

What it can do

Microsoft reports near state-of-the-art results among models under 13 billion parameters on benchmarks covering common sense, language understanding and logical reasoning. It works best with question-and-answer prompts, dialogue transcripts and code prompts. Most of its code training is Python with common standard packages, so other languages and libraries need manual checking. It can state false facts, write broken code, ignore detailed instructions, mishandle informal English or other languages, and repeat social biases or harmful text when pushed.

How to use it

Weights are downloadable under the MIT licence. Support landed in the transformers library at version 4.37.0; older versions need trust_remote_code set when loading. The model can be run through the standard AutoModelForCausalLM and AutoTokenizer classes. Microsoft notes an attention overflow problem in FP16 and suggests toggling autocast in the attention forward pass to work around it.

Safety and testing

The release exists partly as a safety research artefact. Microsoft says web pages in the training mix were filtered for safety and educational value, yet the model can still emit toxic output when deliberately prompted and can reflect social biases. The company chose to publish it anyway so the open-source community could work out ways to cut toxicity right after pretraining. It has not been checked for production reliability.

Reception and debate

Microsoft cautions that outputs should be treated as starting points rather than finished answers, and that direct deployment without evaluation is outside the scope of the project. The model tends to ramble after its first answer, a habit the company attributes to a textbook-heavy training mix.

Why it matters

As an openly released small model, Phi-2 was offered to the research community for studying safety and bias at a scale that is cheap to run. It extends the Phi series, which explored how far a compact model can go with carefully chosen training data.

Questions people ask about Phi-2

What is Phi-2 for?

Microsoft built it for research into small models and safety. In practice it handles question answering, chat-style prompts and code generation, especially Python. The company says its outputs are starting points and should be checked before use.

How big is it and what was it trained on?

It has 2.7 billion parameters and was trained on 250 billion tokens. The mix combined synthetic text from AOAI GPT-3.5 with web data from Falcon RefinedWeb and SlimPajama, filtered using AOAI GPT-4.

Can I download and use it?

Yes. The weights are published under the MIT licence and load through the transformers library from version 4.37.0. Microsoft says production use without your own evaluation is out of scope for the project.

Does it follow instructions like a chatbot?

No. It is a base model with no instruction tuning and no reinforcement learning from human feedback, so it can fail on detailed or subtle instructions and often adds extra text after answering.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on Phi-2 yet, so the facts on this page come from the sources listed above: huggingface.co/microsoft/phi-2. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.