LLaVA
Open-source multimodal chatbot connecting vision encoders to language models.
LLaVA is an open-source large multimodal model that links a vision encoder with a language model for general visual and linguistic tasks. Released in 2023 by researchers including Haotian Liu, it uses GPT-4 generated data for instruction tuning and demonstrates strong performance in multimodal chat and academic benchmarks.
- Connects vision encoders to LLMs for multimodal chat.
- Uses GPT-4 generated data for instruction tuning.
- Achieved 92.53% accuracy on Science QA benchmark.
- Open-source weights under LLAMA 2 Community License.
- Presented as oral paper at NeurIPS 2023.
What it is
LLaVA stands for Large Language and Vision Assistant. It is an end-to-end trained multimodal model that connects a vision encoder to a large language model. The system was introduced in April 2023 by a team including Haotian Liu and Yong Jae Lee. Its primary purpose is to enable general-purpose visual and language understanding through chat interfaces. The model allows users to upload images and ask questions about them, leveraging the reasoning capabilities of underlying language models like LLaMA or Vicuna.
How it came about
The project emerged from research into visual instruction tuning, a method less explored in multimodal fields compared to text-only models. The team presented their first paper in April 2023, detailing the use of GPT-4 to generate instruction-following data for images. This approach aimed to improve zero-shot capabilities on new tasks. The model gained attention for its oral presentation at NeurIPS 2023. Subsequent versions, such as LLaVA-v1.5-13B, were trained in September 2023, refining the architecture and data mixture for better performance.
Inside the model
LLaVA uses an auto-regressive transformer architecture based on LLaMA or Vicuna. It connects a vision encoder to the language model, allowing it to process both images and text. The training process involves instruction tuning on machine-generated data. Specifically, the developers used GPT-4 to create multimodal instruction-following examples. The training dataset includes 558,000 filtered image-text pairs captioned by BLIP, 158,000 GPT-generated instructions, and 450,000 academic task-oriented visual question answering samples. This combination helps the model align visual features with linguistic responses.
What it can do
Early experiments indicate that LLaVA exhibits impressive multimodal chat abilities. It sometimes mimics the behavior of multimodal GPT-4 when presented with unseen images and instructions. On a synthetic multimodal instruction-following dataset, it achieved a relative score of 85.1 per cent compared to GPT-4. When fine-tuned on the Science QA benchmark, the synergy between LLaVA and GPT-4 reached a state-of-the-art accuracy of 92.53 per cent. The model is evaluated on a collection of twelve benchmarks, including five academic visual question answering tests.
How to use it
The model weights, code base, and the GPT-4 generated visual instruction tuning data are publicly available. It is distributed under the LLAMA 2 Community License, which imposes certain conditions on use. The primary intended users are researchers and hobbyists in computer vision, natural language processing, and artificial intelligence. Developers can access the model through platforms like Hugging Face and interact with it via local deployments or community-hosted spaces.
Why it matters
LLaVA represents a significant step in applying instruction tuning to multimodal models. By making its data and code public, it has facilitated further research in large multimodal models. The approach of using powerful language models to generate training data for smaller multimodal systems has influenced subsequent developments in the field. It demonstrates that end-to-end training can yield robust visual and language assistants without requiring massive proprietary datasets.
Questions people ask about LLaVA
What is LLaVA used for?
LLaVA is designed for research on large multimodal models and chatbots. It allows users to engage in visual conversations, where they can upload images and ask questions, enabling general-purpose visual and language understanding.
Is LLaVA open source?
Yes, the model weights, code base, and the visual instruction tuning data generated by GPT-4 are publicly available. It is released under the LLAMA 2 Community License, which permits research and certain commercial uses with conditions.
How was LLaVA trained?
It was trained by fine-tuning LLaMA or Vicuna models on a mix of data. This includes 558,000 image-text pairs, 158,000 GPT-generated multimodal instructions, and 450,000 academic visual question answering samples.
Good reading and sources
There is no English Wikipedia article on LLaVA yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2304.08485, huggingface.co/liuhaotian/llava-v1.5-13b. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.