Token
The unit of text a model reads and writes: a whole word, a piece of a word, a punctuation mark or a byte.
Also: tokens
Models do not see letters or words directly. A tokenizer cuts text into tokens, each mapped to a number from a fixed vocabulary. Common English words are usually one token, while rare words are split into several pieces, so "unbelievable" might become "un", "belie" and "vable". As a rough guide, 1,000 tokens of English is about 750 words. Context windows, prices and speeds are all measured in tokens. Languages with less training text, including many Indian languages, often need more tokens for the same sentence.