Multimodal model
A model that takes in, and sometimes produces, more than one kind of data: text, images, audio or video.
Also: multimodality
Early language models handled text only. Multimodal models add other senses. Some attach a separate image or audio encoder to a language model; others, like GPT-4o in 2024 and the Gemini family from 2023, were trained from the start on several kinds of data together. They can read charts, describe photos, transcribe and translate speech, follow a video, and in some cases speak or draw. Mixing modalities raises new evaluation and safety questions, such as instructions hidden in images.