Tokenization for LLM Engineers: BPE, WordPiece, Unigram and What They Cost You
Every large language model reads and writes tokens, not words. The tokenizer decides how text becomes those tokens, and with them how much a request costs, how much fits in the context…