News
EveryLLM

Text

Reads and writes text. 471 models on EveryLLM so far.

Text is the base modality of every language model: it reads a prompt and writes a reply, token by token. A tokenizer first splits text into tokens, which are whole words, pieces of words or single characters; in English a token is roughly three-quarters of a word. Everything else a model does, from code to reasoning to calling tools, is ultimately expressed as text that it reads or writes.

Tokens and context

Most models use byte-pair encoding, which builds a vocabulary of frequent character sequences. Vocabularies have grown from about 50,000 tokens in GPT-2 to around 200,000 in GPT-4o and 256,000 in Gemma. The context window is how many tokens the model can consider at once: GPT-3 handled 2,048 in 2020, while Gemini 1.5 Pro offered one million in 2024. Languages in non-Latin scripts often need more tokens for the same meaning.

What text-only still means

Many models remain text-only, especially smaller ones and older families such as GPT-3, Llama 2 and BERT, and text still carries most business uses: drafting, summarising, pulling fields out of documents, translation and search. Text-only models are cheaper to train and serve, and easier to evaluate, since their inputs and outputs can be checked and compared directly.

Famous text models

Further reading

ELECTRA WorldELECTRA A pre-training method that trains text encoders to spot replaced tokens.Research 路 Research only 路 2020 RETRO WorldRETRO A retrieval-enhanced transformer that conditions on chunks from a huge document database.Research 路 Research only 路 2021 fnToolformer 馃嚭馃嚫 United StatesToolformer A language model that learns on its own when to call external tools through APIs.Research 路 Research only 路 2023 imgPaLM-E WorldPaLM-E An embodied multimodal language model that grounds words in robot sensor inputMultimodal 路 Research only 路 2023 Galactica 馃嚭馃嚫 United StatesGalactica A language model trained on scientific text for research tasksDomain 路 Conditional licence 路 2022 imgMM1 WorldMM1 A family of multimodal language models up to 30B parameters, dense and MoE.Multimodal 路 Research only 路 2024 AlexaTM 20B 馃嚭馃嚫 United StatesAlexaTM 20B A 20-billion-parameter multilingual sequence-to-sequence model from Amazon's Alexa team.General 路 Research only 路 2022 Orca 2 馃嚭馃嚫 United StatesOrca 2 A small LLaMA-2-based model trained on synthetic data to reason.Small 路 Conditional licence 路 2023 Phi-2 馃嚭馃嚫 United StatesPhi-2 A 2.7 billion parameter Microsoft research model for QA, chat and codeResearch 路 Open weights InstructGPT 馃嚭馃嚫 United StatesInstructGPT GPT-3 models fine-tuned with human feedback to follow instructionsGeneral 路 API only 路 2022 fnWebGPT 馃嚭馃嚫 United StatesWebGPT A GPT-3 model fine-tuned to answer questions while browsing the webGeneral 路 Research only 路 2021 RecurrentGemma 馃嚭馃嚫 United StatesRecurrentGemma Open language models using Google's Griffin recurrent-plus-attention architectureGeneral 路 Open weights 路 2024 Med-PaLM 馃嚭馃嚫 United StatesMed-PaLM A medical-domain language model aligned from Flan-PaLM using instruction prompt tuning.Domain 路 Research only 路 2022 Switch Transformer WorldSwitch Transformer A sparse mixture-of-experts language model scaling to a trillion parameters.Research 路 Research only 路 2021 GLaM 馃嚭馃嚫 United StatesGLaM A trillion-parameter mixture-of-experts language model from Google ResearchResearch 路 Research only 路 2021 Minerva 馃嚭馃嚫 United StatesMinerva A language model trained further on technical content to solve maths and science problemsResearch 路 Research only 路 2022 ALBERT 馃嚭馃嚫 United StatesALBERT A slimmed-down BERT encoder that shares weights across its layersEncoder 路 Open weights 路 2019 imgvidFlamingo 馃嚞馃嚙 United KingdomFlamingo A visual language model family that learns new image and video tasks from a few examples.Multimodal 路 Research only 路 2022 Gopher 馃嚞馃嚙 United KingdomGopher A 280 billion parameter Transformer language model from DeepMind, studied for how scale changes ability.Research 路 Research only 路 2021 RoBERTa 馃嚭馃嚫 United StatesRoBERTa A robustly optimised BERT pretraining study with released models and codeEncoder 路 Open weights 路 2019

The weekly digest

Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.