Tokenizer

Tokenizers are a completely separate stage of the LLM pipeline: they have their own training set, training algorithm (Byte Pair Encoding), and after training implement two functions: encode() from strings to tokens, and…

https://pbs.twimg.com/media/GGzDVPMasAAtLs0.jpg