SentencePiece
A tokenizer library from Google that treats text as a raw stream, spaces included, so it works for languages written without spaces.
SentencePiece, released by Taku Kudo and John Richardson at Google in 2018, learns a subword vocabulary straight from raw text without first splitting it into words. Spaces are kept as a visible symbol, so Japanese, Chinese and Thai, which do not separate words with spaces, are handled the same way as English. It supports both byte-pair encoding and the unigram method. T5, the original LLaMA, Gemma and many multilingual and Indic models use SentencePiece tokenizers, often with vocabularies extended to cover more scripts.