News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
imgwavPhi-4-multimodal

Phi-4-multimodal

A small open model that reads text, images and audio and replies in text

Phi-4-multimodal-instruct is a lightweight open multimodal model from Microsoft, released in 2025. It takes text, image and audio input and produces text, with a 128K token context window, and it was trained across all three modalities in one network rather than chaining separate models together.

Share on WhatsApp

  • One network processes text, images and audio, with a 128K token context window
  • Ranked first on the Hugging Face OpenASR leaderboard at 6.14% word error rate (March 2025)
  • Microsoft calls it the first open model able to summarise speech
  • Averaged 72.2 across four vision-speech benchmarks with synthetic speech queries
  • Text supports 24 languages; audio covers eight

What it is

Phi-4-multimodal-instruct is a lightweight open multimodal foundation model built by Microsoft. It accepts text, images and audio and returns text, working within a 128K token context window. It draws on the language, vision and speech research and datasets behind the Phi-3.5 and Phi-4.0 models. Microsoft positions it for broad commercial and research use, especially where memory or compute is tight, and highlights reasoning in maths and logic, optical character recognition, chart and table reading, and summarising multiple images or video clips. The company says it is meant as a building block for generative AI features.

How it came about

The release grew out of feedback on the Phi-3 series. Previously, users who wanted speech input had to chain two models: one to transcribe audio into text, then a separate language or vision model. That pipeline lost information such as multiple speakers, background noise and the joint alignment of speech, vision and language. With Phi-4-multimodal-instruct, Microsoft trained a single open model across text, vision and audio so every input passes through the same neural network. The company published a blog post and a technical report alongside the release, and made playgrounds available on several platforms.

Inside the model

One network handles text, images and audio together, so speech, vision and language signals sit in the same representation space instead of being handled by separate components. Microsoft says the model uses a new architecture and a larger vocabulary for efficiency, adds multilingual and multimodal support, and applies better post-training for instruction following and function calling. Extra data was added for key multimodal skills. The enhancement process combined supervised fine-tuning, direct preference optimisation and reinforcement learning from human feedback, which the company says supports precise instruction adherence and safety.

What it can do

Microsoft reports strong automatic speech recognition and speech translation, exceeding the expert WhisperV3 and SeamlessM4T-v2-Large models. It ranked first on the Hugging Face OpenASR leaderboard with a word error rate of 6.14% against 6.5% for the next best model as of 4 March 2025. The company describes it as the first open model able to summarise speech, with results close to GPT-4o. On speech question answering it trails Gemini-1.5-Flash and GPT-4o-realtime-preview. With synthetic speech queries over charts, tables and documents it averaged 72.2 across four benchmarks, ahead of InternOmni-7B and several Gemini models.

How to use it

The weights are published openly on Hugging Face as Phi-4-multimodal-instruct, with an ONNX variant for efficient deployment. Microsoft also offers it through Azure, the GitHub Models marketplace, Nvidia and Hugging Face playgrounds, plus demo spaces such as Thoughts Organizer, Stories Come Alive and Phine Speech Translator. A cookbook and a Phi portal are provided for developers.

Safety and testing

Post-training includes direct preference optimisation and reinforcement learning from human feedback, which Microsoft says supports safety measures alongside instruction adherence. The model card tells developers to assess accuracy, safety and fairness before deploying, particularly in high-risk settings, and to follow relevant laws including privacy and trade compliance rules. It notes the model was not evaluated for every downstream purpose and that performance varies across languages.

Reception and debate

Microsoft frames the release as a response to user feedback from the Phi-3 series and says the enthusiastic support for Phi-4 is appreciated, inviting further comments to guide future iterations. The company acknowledges a gap on speech question answering against close rivals and says work is under way to improve it in later versions.

Why it matters

By folding speech, vision and language into one open network, the model removes the two-stage transcription pipeline that earlier Phi users relied on. Microsoft says this should help app developers whose systems need to observe several speakers, background noise and aligned audio-visual-language information at once, and it is offered as a research building block for generative AI features.

Questions people ask about Phi-4-multimodal

What does Phi-4-multimodal actually do?

It accepts text, images and audio as input and produces text as output, all through a single neural network. Microsoft built it for tasks such as speech recognition, speech translation, speech summarisation, optical character recognition, chart and table understanding, and summarising images or video clips.

How is it different from earlier Phi models?

With Phi-3, speech input required two models chained together: one transcribed audio to text, then another handled language or vision. Phi-4-multimodal-instruct is trained across text, vision and audio at once, so it can see multiple speakers, background noise and aligned signals in one representation space.

How do I get it?

The instruct weights are openly available on Hugging Face, along with an ONNX version. Microsoft also runs it through Azure, the GitHub Models marketplace, Nvidia and Hugging Face playgrounds, and provides a cookbook and demo spaces for developers.

What languages does it cover?

Text input spans 24 languages including Arabic, Chinese, English, French, German, Japanese, Korean, Russian, Spanish and Thai. Audio input covers eight: English, Chinese, German, French, Italian, Japanese, Spanish and Portuguese.

What are its limits?

Microsoft says it lags Gemini-1.5-Flash and GPT-4o-realtime-preview on speech question answering, and notes performance differs across languages. The company says it was not evaluated for every downstream use and asks developers to check accuracy, safety and fairness before deploying, especially in high-risk settings.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on Phi-4-multimodal yet, so the facts on this page come from the sources listed above: huggingface.co/microsoft/Phi-4-multimodal-instruct. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.