News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
imgMM1

MM1

A family of multimodal language models up to 30B parameters, dense and MoE.

MM1 is a family of multimodal large language models, up to 30 billion parameters, in both dense and mixture-of-experts forms. It was built by a team of 32 researchers led by Brandon McKinzie and Zhe Gan, and described in a paper first posted on 14 March 2024. The work studies how architecture and training data choices affect multimodal pre-training.

Share on WhatsApp

  • Family of multimodal models up to 30B parameters, dense and mixture-of-experts
  • First posted as an arXiv paper on 14 March 2024
  • Image encoder, resolution and image token count strongly affect results
  • Vision-language connector design was found to matter comparatively little
  • Mixing captioned, interleaved and text-only data was crucial for few-shot results

What it is

MM1 is a family of multimodal large language models, meaning models that handle both images and text. The largest version reaches 30 billion parameters, and the family includes dense models alongside mixture-of-experts variants, where only part of the network is active for each token. The work was published as a research paper first posted on 14 March 2024, with a revised version in April 2024. Its purpose is to show which design choices matter most when pre-training models at scale on mixed image and text data, and to show what such a recipe can achieve on established multimodal benchmarks.

How it came about

The paper was written by 32 authors, with Brandon McKinzie and Zhe Gan listed first. It appeared on arXiv on 14 March 2024 and was revised several times, with the fourth version dated 18 April 2024. The team reports that scaling up their recipe produced the MM1 family. The work sits in the computer vision, computation and language, and machine learning research areas.

Inside the model

The team ran careful ablations, meaning controlled comparisons, across three parts of the system: the image encoder, the connector that joins vision to language, and the pre-training data mix. They found the image encoder, its resolution, and the number of image tokens used all have a large effect on results. The vision-language connector design mattered comparatively little. For data, a deliberate blend of captioned images, interleaved image and text documents, and plain text was crucial for strong few-shot results. The family spans dense and mixture-of-experts designs up to 30B parameters.

What it can do

The authors report state-of-the-art pre-training metrics against other published pre-training results, and competitive performance after supervised fine-tuning on established multimodal benchmarks. Large-scale pre-training gave the models in-context learning and multi-image reasoning, which allows few-shot chain-of-thought prompting, where the model is shown worked examples and reasons step by step. No specific benchmark scores are given in the available description.

How to use it

The work is presented as a research paper on arXiv, with a PDF available. No release of downloadable weights, licence terms, app or API is described.

Why it matters

MM1 is presented as a study of what matters in multimodal pre-training, offering design lessons on encoders, connectors and data mixing that others building image-and-text models can follow. Its ablations isolate which components drive performance at scale.

Questions people ask about MM1

What is MM1?

MM1 is a family of multimodal large language models that take images and text together, described in a research paper first posted in March 2024. The models go up to 30 billion parameters and come in both dense and mixture-of-experts versions. The paper focuses on which design and data choices matter most during pre-training.

Who created it?

A team of 32 researchers, with Brandon McKinzie and Zhe Gan as the first two authors. The paper was posted on arXiv on 14 March 2024 and revised in April 2024. The available description does not name the organisation behind the work.

What did the researchers find matters most?

The image encoder, the resolution of the images, and how many image tokens are used all had a substantial impact on performance. The design of the connector between vision and language mattered comparatively little. For data, a careful mix of captioned images, interleaved image-text, and plain text was crucial for strong few-shot results.

What can MM1 do?

The authors report state-of-the-art pre-training metrics compared with other published pre-training results, and competitive scores after supervised fine-tuning on established multimodal benchmarks. Large-scale pre-training also gave the models in-context learning and multi-image reasoning, supporting few-shot chain-of-thought prompting.

Can I download or use MM1?

The work is available as a research paper with a PDF on arXiv. The description does not cover released weights, a licence, an app or an API, so no access route beyond the paper itself is described.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on MM1 yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2403.09611. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.