Vocabulary
The fixed list of tokens a model knows, each with its own number and its own embedding.
Also: vocab
A model's vocabulary is the complete set of tokens its tokenizer can produce. GPT-2 used about 50,000 tokens, Llama 3 uses about 128,000, and Gemma about 256,000. A larger vocabulary means common words and words in more languages fit in fewer tokens, which makes text shorter and inference cheaper, but it also makes the embedding table and the final output layer bigger. Vocabularies also hold special tokens that mark the start and end of a text or separate the turns of a chat.