Byte-pair encoding
A method for building a vocabulary by repeatedly merging the most frequent pair of symbols in a body of text.
Also: BPE
Byte-pair encoding began as a data compression method, described by Philip Gage in 1994. In 2016 Rico Sennrich and colleagues adapted it for machine translation, and GPT-2 and most later models use a byte-level version. Training starts with single characters or bytes and keeps merging the most common neighbouring pair into a new symbol until the vocabulary reaches its target size. Frequent words end up as one token and rare words as several. Because it works on bytes, a byte-level BPE tokenizer can encode any text, emoji included, without unknown symbols.