Tokenizer
The program that splits text into tokens and turns them into numbers before a model sees them, and back again afterwards.
Also: tokeniser, tokenisation
A tokenizer is fixed before training and stays with the model for life. It holds a vocabulary of tens of thousands to a few hundred thousand pieces and a set of rules for splitting text into them. Most modern tokenizers are built with byte-pair encoding or the related unigram method, learned from a sample of the training data. The choice matters: a tokenizer trained mostly on English splits Hindi, Tamil or Arabic into many small pieces, which makes the model slower and more expensive in those languages.