What they take in and give out
The kinds of input and output a model handles.
Text
Text is the base modality of every language model: it reads a prompt and writes a reply, token by token. A tokenizer first splits text into tokens, which are whole words, pieces of words or single characters; in English a token is roughly three-quarters of a word
Images in
Models with image input can read photographs, screenshots, charts, diagrams, handwriting and scanned documents, and answer questions about them. An image encoder, often a vision transformer, cuts the picture into small patches and turns each into a vector that the language model treats like a token
Speech and audio
Speech and audio models listen and speak. Older voice assistants chained three systems: speech recognition to turn speech into text, a language model, and text-to-speech
Video in
Models with video input watch clips and answer questions about them: what happens, when, and what is said. Video is the most demanding input, because even a short clip contains thousands of frames, so models usually sample about one frame per second, turn each into image tokens and add the soundtrack as audio or as a transcript.
Images out
Some language models draw. Instead of handing a prompt to a separate image generator, a natively multimodal model can produce image tokens itself, which lets it follow detailed instructions, render legible text inside pictures and edit an image through conversation, keeping faces and objects consistent from one turn to the next.
Tools and agents
Tool use lets a model act rather than only answer. The model writes a structured request, such as a search query, a function call with arguments or a command; the surrounding software runs it; and the result comes back for the model to read