News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
imgCogVLM

CogVLM

Open-source visual language model with deep feature fusion

CogVLM is an open-source visual language model released in late 2023 by researchers including Weihan Wang and Jie Tang. It uses a unique visual expert module to fuse image and text data deeply, achieving top scores on ten cross-modal benchmarks while maintaining strong natural language processing abilities.

Share on WhatsApp

  • Contains 10 billion vision and 7 billion language parameters
  • Uses a trainable visual expert module for deep feature fusion
  • Achieves state-of-the-art scores on 10 cross-modal benchmarks
  • Code is open source under Apache-2.0 license
  • Requires approximately 40GB GPU VRAM for inference

What it is

CogVLM is a visual language foundation model designed to process both images and text. Released in November 2023, the primary version contains 17 billion parameters, split into 10 billion for vision tasks and 7 billion for language. The model aims to bridge the gap between frozen pretrained language models and image encoders without losing performance on standard text tasks. It is developed by a team of researchers associated with THUDM.

How it came about

The model was introduced in a paper submitted to arXiv in November 2023, with a revised version appearing in February 2024. The development team, led by authors such as Weihan Wang and Jie Tang, focused on creating a system that could match or surpass larger models like PaLI-X 55B. The project made its code and checkpoints publicly available to support further research and commercial use under specific conditions.

Inside the model

Unlike methods that simply map image features into the language model input space, CogVLM employs a trainable visual expert module. This component sits within the attention and feed-forward network layers, allowing for deep fusion of visual and linguistic features. The architecture comprises four main parts: a vision transformer encoder, an MLP adapter, a pretrained large language model, and the visual expert module. This design ensures that natural language processing capabilities remain intact while adding visual understanding.

What it can do

The model achieves state-of-the-art results on ten classic cross-modal benchmarks, including NoCaps, Flicker30k captioning, RefCOCO, and ScienceQA. It also ranks second on tests such as VQAv2, OKVQA, and COCO captioning. In these evaluations, it performs comparably to or better than PaLI-X 55B, despite having fewer total parameters. The system can handle tasks like visual question answering and image captioning effectively.

How to use it

CogVLM is an open-source project. The underlying code is released under the Apache-2.0 license. Use of the model weights requires compliance with a separate model license, which allows for free commercial use after registering via a form. Researchers and developers can download the weights from platforms like Hugging Face. Running the model requires significant hardware resources, specifically around 40GB of GPU memory for inference.

Questions people ask about CogVLM

What makes CogVLM different from other visual language models?

CogVLM uses a trainable visual expert module within the attention and FFN layers. This allows for deep fusion of vision and language features, unlike shallow alignment methods that just map image features into the input space.

How much computer memory do I need to run CogVLM?

You need nearly 40GB of GPU VRAM for inference. If you lack a single GPU with this capacity, you can use the accelerate library to split the model across multiple GPUs with smaller memory sizes.

Can I use CogVLM for commercial projects?

Yes, but you must follow the model license. While the code is Apache-2.0 licensed, using the weights for commercial purposes requires filling out a registration form provided by the developers.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on CogVLM yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2311.03079, huggingface.co/zai-org/cogvlm-chat-hf. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.