Video in
Watches video. 36 models on EveryLLM so far.
Models with video input watch clips and answer questions about them: what happens, when, and what is said. Video is the most demanding input, because even a short clip contains thousands of frames, so models usually sample about one frame per second, turn each into image tokens and add the soundtrack as audio or as a transcript.
Long context made it practical
Video understanding became practical when context windows grew. Google's Gemini 1.5 Pro (February 2024), with a window of one million tokens, could take in about an hour of video and find a moment within it. Alibaba's Qwen2.5-VL (January 2025) and Moonshot AI's Kimi models accept video in open-weights form. Live video lets assistants see through a phone camera and comment in real time.
Watching and making
Video in is different from video out. Text-to-video generators such as OpenAI's Sora and Google's Veo are mostly diffusion models rather than language models, but some systems join the two: Google's VideoPoet (2023) used an autoregressive language model to generate video from text, images and video. Robotics models such as Gemini Robotics take a stream of camera images and turn it into actions.
Famous video in models
Still to be written: GPT-4o.
Further reading
Nothing matches those filters yet. Take one off, or check back: new pages are written every day.