News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
Orca 2

Orca 2

A small LLaMA-2-based model trained on synthetic data to reason.

Orca 2 is a small language model fine-tuned from LLaMA-2 by Microsoft researchers to reason over data, read, solve maths and summarise. Released in November 2023 with public weights for research, it showed that synthetic training data can teach small models reasoning strategies usually seen in far larger systems.

Share on WhatsApp

  • Fine-tuned from LLaMA-2 base, 13 billion parameters
  • Trained on synthetic data teaching several reasoning strategies
  • Evaluated on 15 benchmarks, about 100 tasks, 36,000+ prompts
  • Reported to match models 5-10x larger on zero-shot reasoning
  • Research-only weights under the Microsoft Research License

What it is

Orca 2 is a small language model built by Microsoft researchers as a research artefact. It is a fine-tuned version of the LLaMA-2 base model, released in November 2023. The team says it handles single-turn tasks such as reasoning over user-supplied data, reading comprehension, maths problem solving and summarisation, with reasoning as its strongest focus. The published 13-billion-parameter weights are meant for research on how smaller models develop, evaluate and align reasoning skills, not for chat or production use. The work continues an earlier model called Orca 1, which learned from rich signals such as explanation traces.

How it came about

The project follows Orca 1, which used explanation traces and beat conventional instruction-tuned models on benchmarks including BigBench Hard and AGIEval. For Orca 2, a team of fifteen Microsoft researchers argued that copying a larger model's answers can hold small models back. Their paper appeared on arXiv on 18 November 2023, revised three days later to add the weights link. The weights were published openly to support research on smaller language models.

Inside the model

Orca 2 starts from the LLaMA-2 base model and is fine-tuned on a synthetic dataset built to strengthen reasoning. Rather than imitating one teacher's answers, the training teaches several solution strategies: step-by-step work, recall then generate, recall-reason-generate, and direct answers. The aim is for the model to pick the strategy that suits each task. Training data was filtered through Microsoft Azure content filters. The model was not trained with RLHF or DPO, so it is not tuned for chat, and the team recommends fine-tuning it for chat or a specific task first.

What it can do

Evaluation covered 15 benchmarks, roughly 100 tasks and more than 36,000 unique prompts, spanning reasoning, grounding and safety. In zero-shot settings on complex reasoning tasks, the authors report that Orca 2 clearly beats similar-sized models and reaches levels comparable to or better than models five to ten times larger. The gains do not carry over to few-shot prompting, where larger models still benefit more. Performance also varies with system instructions, and accuracy may drop in areas underrepresented in the tuning data, such as maths and coding.

How to use it

Weights are publicly available under the Microsoft Research License, with LLaMA 2 itself under Meta's LLAMA 2 Community License. The model card gives Hugging Face inference code and recommends the slow tokenizer. Microsoft positions it as research-only and suggests pairing it with Azure AI Content Safety for moderation.

Safety and testing

All synthetic training data was run through Microsoft Azure content filters. The model card lists risks including data bias, weak real-world understanding, opaque outputs, content harms, hallucination and possible misuse for disinformation. It notes that synthetic training may pass on shortcomings of the data-generation process, and that small models might hallucinate more in ungrounded settings, though this is described as an open research question. Testing was confined to research settings, and the card advises against downstream deployment without further analysis.

Reception and debate

The model card states plainly that Orca 2 is for research only and should not be used in downstream applications. It also warns that results depend heavily on the tuning data distribution and on the system message given at inference time.

Why it matters

Orca 2 was released to support research on developing, evaluating and aligning smaller language models. Its central claim is that capable models and complex workflows, using advanced prompts and multiple calls, can generate synthetic data that teaches small models new abilities. The team says the Orca training approach has already been applied successfully to base models other than LLaMA-2.

Questions people ask about Orca 2

What is Orca 2 for?

It is a research model for studying how small language models learn to reason. Microsoft says it handles single-turn tasks like reasoning over given data, reading comprehension, maths and summarisation, and that it should not be used in downstream applications without further analysis.

How does it differ from Orca 1?

Orca 1 learned from rich signals such as explanation traces. Orca 2 goes further by teaching a small model several solution strategies and how to choose between them, instead of simply copying the answers of a larger teacher model.

Can I chat with it?

Not out of the box. The model was not trained with RLHF or DPO and is not optimised for chat. Microsoft recommends fine-tuning it for chat or a particular task before use, and notes its behaviour varies with the system message.

How is it licensed?

The Orca 2 weights are released under the Microsoft Research License. The underlying LLaMA 2 model remains under Meta's LLAMA 2 Community License.

What are its main limitations?

The model card flags bias from training data, limited real-world understanding, hallucination, and accuracy that tracks the tuning data distribution, so maths, coding and reasoning may be weaker where data is thin. Few-shot prompting brings smaller gains than in larger models.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on Orca 2 yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2311.11045, huggingface.co/microsoft/Orca-2-13b. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.