MM1
A family of multimodal language models up to 30B parameters, dense and MoE.
MM1 is a family of multimodal large language models, up to 30 billion parameters, in both dense and mixture-of-experts forms. It was built by a team of 32 researchers led by Brandon McKinzie and Zhe Gan, and described in a paper first posted on 14 March 2024. The work studies how architecture and training data choices affect multimodal pre-training.
- Family of multimodal models up to 30B parameters, dense and mixture-of-experts
- First posted as an arXiv paper on 14 March 2024
- Image encoder, resolution and image token count strongly affect results
- Vision-language connector design was found to matter comparatively little
- Mixing captioned, interleaved and text-only data was crucial for few-shot results
What it is
MM1 is a family of multimodal large language models, meaning models that handle both images and text. The largest version reaches 30 billion parameters, and the family includes dense models alongside mixture-of-experts variants, where only part of the network is active for each token. The work was published as a research paper first posted on 14 March 2024, with a revised version in April 2024. Its purpose is to show which design choices matter most when pre-training models at scale on mixed image and text data, and to show what such a recipe can achieve on established multimodal benchmarks.
How it came about
The paper was written by 32 authors, with Brandon McKinzie and Zhe Gan listed first. It appeared on arXiv on 14 March 2024 and was revised several times, with the fourth version dated 18 April 2024. The team reports that scaling up their recipe produced the MM1 family. The work sits in the computer vision, computation and language, and machine learning research areas.
Inside the model
The team ran careful ablations, meaning controlled comparisons, across three parts of the system: the image encoder, the connector that joins vision to language, and the pre-training data mix. They found the image encoder, its resolution, and the number of image tokens used all have a large effect on results. The vision-language connector design mattered comparatively little. For data, a deliberate blend of captioned images, interleaved image and text documents, and plain text was crucial for strong few-shot results. The family spans dense and mixture-of-experts designs up to 30B parameters.
What it can do
The authors report state-of-the-art pre-training metrics against other published pre-training results, and competitive performance after supervised fine-tuning on established multimodal benchmarks. Large-scale pre-training gave the models in-context learning and multi-image reasoning, which allows few-shot chain-of-thought prompting, where the model is shown worked examples and reasons step by step. No specific benchmark scores are given in the available description.
How to use it
The work is presented as a research paper on arXiv, with a PDF available. No release of downloadable weights, licence terms, app or API is described.
Why it matters
MM1 is presented as a study of what matters in multimodal pre-training, offering design lessons on encoders, connectors and data mixing that others building image-and-text models can follow. Its ablations isolate which components drive performance at scale.
Questions people ask about MM1
What is MM1?
MM1 is a family of multimodal large language models that take images and text together, described in a research paper first posted in March 2024. The models go up to 30 billion parameters and come in both dense and mixture-of-experts versions. The paper focuses on which design and data choices matter most during pre-training.
Who created it?
A team of 32 researchers, with Brandon McKinzie and Zhe Gan as the first two authors. The paper was posted on arXiv on 14 March 2024 and revised in April 2024. The available description does not name the organisation behind the work.
What did the researchers find matters most?
The image encoder, the resolution of the images, and how many image tokens are used all had a substantial impact on performance. The design of the connector between vision and language mattered comparatively little. For data, a careful mix of captioned images, interleaved image-text, and plain text was crucial for strong few-shot results.
What can MM1 do?
The authors report state-of-the-art pre-training metrics compared with other published pre-training results, and competitive scores after supervised fine-tuning on established multimodal benchmarks. Large-scale pre-training also gave the models in-context learning and multi-image reasoning, supporting few-shot chain-of-thought prompting.
Can I download or use MM1?
The work is available as a research paper with a PDF on arXiv. The description does not cover released weights, a licence, an app or an API, so no access route beyond the paper itself is described.
Good reading and sources
There is no English Wikipedia article on MM1 yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2403.09611. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.