NVLM 1.0
Open multimodal language models from NVIDIA for vision and language tasks
NVLM 1.0 is a family of multimodal large language models announced by NVIDIA in September 2024. The team reports results on vision-language tasks that match leading proprietary and open-access models, while keeping or improving text-only ability compared with the language model backbones they were built from.
- Family of multimodal LLMs from NVIDIA, first posted 17 September 2024
- Team reports results rivaling GPT-4o, Llama 3-V 405B and InternVL 2
- Text-only ability kept or improved after multimodal training, the authors say
- New 1-D tile-tagging scheme for tiled high-resolution images
- Model weights released; training code promised for the community
What it is
NVLM 1.0 is a family of multimodal large language models, meaning models that handle both images and text. NVIDIA researchers introduced it in a paper first posted in September 2024. The models are built on top of existing text-only language model backbones and then trained further with image and text data. The team says the family reaches results on vision-language tasks that compete with leading proprietary systems such as GPT-4o and with open-access models including Llama 3-V 405B and InternVL 2. A notable claim is that text-only ability is preserved, and sometimes improved, after this multimodal training.
How it came about
The work was carried out by a team of ten NVIDIA researchers, including Wenliang Dai, Nayeon Lee, Boxin Wang and Wei Ping. The paper appeared on arXiv on 17 September 2024, with a revised version on 22 October 2024 that the authors say fixed typographical errors. Alongside the paper, the team released model weights and said training code would follow for the research community.
Inside the model
The team compared two broad designs: decoder-only multimodal models in the style of LLaVA, and cross-attention models in the style of Flamingo. Weighing the strengths of each, they proposed a hybrid architecture intended to improve both training efficiency and multimodal reasoning. For high-resolution images, they use tiling and add a 1-D tile-tagging scheme, which they say helps reasoning and OCR tasks. Training mixes curated multimodal pretraining and supervised fine-tuning data with a high-quality text-only set plus maths and reasoning examples. They report that data quality and task diversity matter more than sheer scale.
What it can do
The models are aimed at vision-language work, including OCR and multimodal reasoning. The authors report top-tier results on such tasks, comparing their systems with GPT-4o, Llama 3-V 405B and InternVL 2. They also claim stronger maths and coding performance across modalities, and text-only performance that at least matches the underlying language model backbones. Specific benchmark figures are not given in the summary provided here.
How to use it
Model weights were released publicly at the time of publication. The authors stated that training code would be open-sourced for the community later. No separate licence terms are described here.
Why it matters
The release is presented as a contribution to open research on multimodal models, with weights made available and code promised. Its architecture comparison and the 1-D tile-tagging method for high-resolution images are offered as design guidance for others building vision-language systems.
Questions people ask about NVLM 1.0
What is NVLM 1.0?
It is a family of multimodal large language models from NVIDIA, introduced in September 2024. They take in both images and text and are designed for vision-language tasks such as OCR and multimodal reasoning, while retaining text-only ability.
Who made it and when?
A team of ten NVIDIA researchers, led on the paper by Wenliang Dai and colleagues. The first version of the paper was posted on 17 September 2024, with a revision on 22 October 2024.
How does it differ from other multimodal models?
The team compared decoder-only designs like LLaVA with cross-attention ones like Flamingo, then proposed a hybrid architecture. They also introduced 1-D tile-tagging for tiled high-resolution images, which they say improves reasoning and OCR performance.
Can I use it?
Model weights were released publicly alongside the paper. The authors said training code would be open-sourced for the community. Specific licence terms are not described in the information given here.
Good reading and sources
There is no English Wikipedia article on NVLM 1.0 yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2409.11402. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.