Phi-2
A 2.7 billion parameter Microsoft research model for QA, chat and code
Phi-2 is a compact text-generating Transformer built by Microsoft, with 2.7 billion parameters. Released as an open research model, it targets common sense, language understanding and reasoning tasks, and was put out to let researchers study safety problems such as toxicity and bias in small models.
- 2.7 billion parameters, released openly by Microsoft
- Trained on 250 billion tokens of synthetic text and web data
- No RLHF and no instruction tuning applied
- MIT licence, supported in transformers 4.37.0
- Known FP16 attention overflow issue
What it is
Phi-2 is a language model with 2.7 billion parameters, built on the Transformer design and trained to predict the next word. Microsoft released it openly for research use. It is aimed at everyday reasoning, reading comprehension and code writing, and the company positions it as a strong performer among models below 13 billion parameters on tests of common sense, language understanding and logic. Because it is small, it is cheap to run and easy to study. It is a base model, so it has not been tuned to follow instructions or to chat politely.
How it came about
Phi-2 continues the Phi line from Microsoft, following Phi-1.5. It reused that model's training sources and added a further collection of synthetic language data plus web pages chosen for safety and educational worth. The team published the weights openly so that researchers could examine safety questions on a small, unrestricted model. Microsoft says the goal was to support work on lowering toxicity, understanding social bias and improving controllability.
Inside the model
The system is a Transformer that learns by guessing the next word in a sequence. Its training set totals 250 billion tokens. It combines synthetic text produced by AOAI GPT-3.5 with web corpora drawn from Falcon RefinedWeb and SlimPajama, and those web pages were reviewed by AOAI GPT-4. Training used PyTorch, DeepSpeed and Flash-Attention. No reinforcement learning from human feedback was applied, and there was no instruction tuning, so the raw pretrained behaviour is what users get.
What it can do
Microsoft reports near state-of-the-art results among models under 13 billion parameters on benchmarks covering common sense, language understanding and logical reasoning. It works best with question-and-answer prompts, dialogue transcripts and code prompts. Most of its code training is Python with common standard packages, so other languages and libraries need manual checking. It can state false facts, write broken code, ignore detailed instructions, mishandle informal English or other languages, and repeat social biases or harmful text when pushed.
How to use it
Weights are downloadable under the MIT licence. Support landed in the transformers library at version 4.37.0; older versions need trust_remote_code set when loading. The model can be run through the standard AutoModelForCausalLM and AutoTokenizer classes. Microsoft notes an attention overflow problem in FP16 and suggests toggling autocast in the attention forward pass to work around it.
Safety and testing
The release exists partly as a safety research artefact. Microsoft says web pages in the training mix were filtered for safety and educational value, yet the model can still emit toxic output when deliberately prompted and can reflect social biases. The company chose to publish it anyway so the open-source community could work out ways to cut toxicity right after pretraining. It has not been checked for production reliability.
Reception and debate
Microsoft cautions that outputs should be treated as starting points rather than finished answers, and that direct deployment without evaluation is outside the scope of the project. The model tends to ramble after its first answer, a habit the company attributes to a textbook-heavy training mix.
Why it matters
As an openly released small model, Phi-2 was offered to the research community for studying safety and bias at a scale that is cheap to run. It extends the Phi series, which explored how far a compact model can go with carefully chosen training data.
Questions people ask about Phi-2
What is Phi-2 for?
Microsoft built it for research into small models and safety. In practice it handles question answering, chat-style prompts and code generation, especially Python. The company says its outputs are starting points and should be checked before use.
How big is it and what was it trained on?
It has 2.7 billion parameters and was trained on 250 billion tokens. The mix combined synthetic text from AOAI GPT-3.5 with web data from Falcon RefinedWeb and SlimPajama, filtered using AOAI GPT-4.
Can I download and use it?
Yes. The weights are published under the MIT licence and load through the transformers library from version 4.37.0. Microsoft says production use without your own evaluation is out of scope for the project.
Does it follow instructions like a chatbot?
No. It is a base model with no instruction tuning and no reinforcement learning from human feedback, so it can fail on detailed or subtle instructions and often adds extra text after answering.
Good reading and sources
There is no English Wikipedia article on Phi-2 yet, so the facts on this page come from the sources listed above: huggingface.co/microsoft/phi-2. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.