Speech and audio
Listens and speaks. 42 models on EveryLLM so far.
Speech and audio models listen and speak. Older voice assistants chained three systems: speech recognition to turn speech into text, a language model, and text-to-speech. Newer models are trained on audio directly, so they hear tone and hesitation, can reply in natural speech within a fraction of a second, and let users interrupt them mid-sentence.
Listening
OpenAI's Whisper, released as open source in September 2022, was trained on 680,000 hours of multilingual audio from the web and became a standard tool for transcription. Language models now accept audio directly: Gemini models take long recordings, and Alibaba's Qwen2.5-Omni (March 2025) reads text, images, audio and video in one model. In India, Sarvam AI and the government's Bhashini programme focus on speech in Indian languages.
Speaking
GPT-4o, shown in May 2024, answered spoken questions in about 320 milliseconds on average, close to human turn-taking, and ChatGPT's Advanced Voice Mode reached paying users from September 2024. In July 2024 the Paris lab Kyutai demonstrated Moshi, a model that listens and speaks at the same time, and later released it openly. Voice cloning raises fraud risks, so most providers limit the choice of voices.
Famous speech and audio models
Still to be written: GPT-4o.