Tokenization in LLMs: Why It Matters More Than You Think

An LLM never sees a word. Before your prompt reaches the model it is cut into tokens, and each token is replaced by a number. The model reads numbers, predicts numbers, and the result is turned back into text at the end.

Tokenization in LLMs is easy to ignore, because the interface hides it. It is worth understanding for three reasons: you are billed by the token, limits are set in tokens, and several odd behaviours of models are really behaviours of tokens.

What a token is

A token is a piece of text that the model treats as one unit. It can be a whole word, part of a word, a punctuation mark or a single byte. OpenAI's documentation describes tokens as commonly occurring sequences of characters, and gives a rule of thumb for English: one token is about four characters, or three quarters of a word (OpenAI: key concepts).

Other vendors give similar figures for their own models: about 4 characters for Gemini, with 100 tokens equal to roughly 60 to 80 English words (Google: understand and count tokens), and about 3.5 English characters for Claude (Anthropic: glossary). The figures differ because each family of models has its own tokenizer.

Why not words or letters

Subword tokenization keeps frequent words as single tokens and breaks rare ones into parts it already knows, so nothing is ever out of vocabulary (Hugging Face: tokenization algorithms).

How the vocabulary is built

The most common method is byte pair encoding, or BPE, which was brought into neural language models to handle rare words (Sennrich et al.: subword units). The recipe is short:

1. Start with the smallest units. In the byte-level form, that is the 256 possible bytes, so any text in any script can be represented.
2. Count which pair of neighbouring units appears most often in a large body of text.
3. Merge that pair into a new unit and add it to the vocabulary.
4. Repeat until the vocabulary reaches the size you want.

GPT-2's vocabulary of 50,257 is exactly that: 256 bytes, 50,000 merges and one special end-of-text token. The result is reversible and lossless, and it compresses: OpenAI's tokenizer library notes that a token corresponds to about four bytes on average (OpenAI: tiktoken).

The merges are learned from data, so what counts as "common" depends on the text the tokenizer was built from. That has consequences, as the measurements show.

Measuring it

The script uses tiktoken, OpenAI's open-source tokenizer. It shows how strings split, then counts the tokens in one sentence written in English, Hindi and Telugu, under three generations of encoding.

import json

import tiktoken

for model in ("gpt2", "gpt-4", "gpt-4o"):
    used = tiktoken.encoding_for_model(model)
    print(f"{model:7} uses the encoding {used.name:12} with {used.n_vocab:,} tokens")

enc = tiktoken.get_encoding("o200k_base")
print()


def pieces(text):
    return " | ".join(enc.decode([t]) for t in enc.encode(text))


for text in ("the", "tokenization", "unbelievably", "Hyderabad", "1234567890", "hello", " hello", "HELLO",
             "df.groupby('city').sum()"):
    print(f"{text!r:28} {len(enc.encode(text))} tokens:  {pieces(text)}")

SENTENCES = {"English": "Data engineering is a good career choice.",
             "Hindi": "डेटा इंजीनियरिंग एक अच्छा करियर विकल्प है।",
             "Telugu": "డేటా ఇంజనీరింగ్ ఒక మంచి కెరీర్ ఎంపిక."}
print()
print("the same sentence     characters   gpt2   cl100k_base   o200k_base")
for language, sentence in SENTENCES.items():
    counts = [len(tiktoken.get_encoding(name).encode(sentence)) for name in ("gpt2", "cl100k_base", "o200k_base")]
    print(f"{language:20} {len(sentence):>11}   {counts[0]:>4}   {counts[1]:>11}   {counts[2]:>10}")

prose = next(s["text"] for s in json.load(open("../../genai/pep_sections.json", encoding="utf-8"))
             if s["id"] == "pep8-2")              # a section of PEP 8 that is plain English prose
words, tokens = len(prose.split()), len(enc.encode(prose))
print()
print(f"{words} words of English prose: {len(prose):,} characters, {tokens} tokens, "
      f"{len(prose) / tokens:.1f} characters and {words / tokens:.2f} words per token")

gpt2    uses the encoding gpt2         with 50,257 tokens
gpt-4   uses the encoding cl100k_base  with 100,277 tokens
gpt-4o  uses the encoding o200k_base   with 200,019 tokens

'the'                        1 tokens:  the
'tokenization'               2 tokens:  token | ization
'unbelievably'               3 tokens:  un | bel | ievably
'Hyderabad'                  2 tokens:  Hy | derabad
'1234567890'                 4 tokens:  123 | 456 | 789 | 0
'hello'                      1 tokens:  hello
' hello'                     1 tokens:   hello
'HELLO'                      2 tokens:  HEL | LO
"df.groupby('city').sum()"   7 tokens:  df | .groupby | (' | city | '). | sum | ()

the same sentence     characters   gpt2   cl100k_base   o200k_base
English                       41      8             8            8
Hindi                         42     70            46           13
Telugu                        37     99            63           16

235 words of English prose: 1,435 characters, 302 tokens, 4.8 characters and 0.78 words per token

• Common words are one token, rarer ones are pieces. "the" is one. "tokenization" is "token" and "ization". "Hyderabad" is "Hy" and "derabad", which is a statistical cut, not a meaningful one.
• Numbers are chopped into groups of digits. "1234567890" is four tokens. The model sees "123", "456", "789" and "0", not a number.
• Capitals and spaces change the result. "hello" is one token and "HELLO" is two. A word with a space in front of it is a different token from the same word without.
• Code is punctuation-heavy. One short line of pandas is seven tokens.
• The same sentence is 8 tokens in English and 99 in Telugu under the oldest encoding, with fewer characters. That tokenizer was built from mostly English text, so other scripts fall back towards single bytes.
• The newest encoding closes most of the gap: 13 tokens for Hindi and 16 for Telugu, against 8 for English.
• The rule of thumb holds for English prose: 4.8 characters and 0.78 words per token here.

What follows

Cost. Input and output are both billed per token. The same request in Hindi or Telugu can cost a multiple of the English one on an older model, and still up to double on a newer one. Measure with the tokenizer for your model, not with a word count. See cost optimization for LLM applications.

Limits. The context window is counted in tokens, and the prompt and the reply share it. Code, tables, JSON and non-English text use it up faster than prose. See context windows explained.

Spelling and counting. A model that receives "Hy" and "derabad" has no direct view of the letters inside. Questions such as "how many times does a letter appear in this word" ask it about something it cannot see, which is why models can get them wrong.

Arithmetic. Long numbers arrive as arbitrary chunks of digits. For exact calculation, give the model a calculator tool. See function calling and tool use.

Chunk sizes. Documents are usually split by token count before being indexed for retrieval, because that is the unit the embedding model and the context window care about. See chunking strategies for RAG.

Other tokenizers

BPE is one of three common algorithms. WordPiece and Unigram build their vocabularies differently, and a tool called SentencePiece applies these methods to raw text without assuming that spaces separate words, which matters for languages such as Chinese and Japanese.

For practical work the rule is simple. A tokenizer belongs to a model. Counts from one are estimates for another, and each vendor's API offers a way to count tokens for its own models. Use that for anything where the number matters.

Practical habits

1. Count before you send, for prompts that are large or run often.
2. Budget for the reply. Output tokens count against the same window.
3. Trim what the model does not need: repeated instructions, whole documents where a section would do.
4. Prefer compact formats for data you pass in. Deeply nested JSON spends tokens on brackets and quotes.
5. Test in the language your users write in. An estimate made in English will be wrong for them.

Common mistakes

• Estimating cost from word counts.
• Assuming one tokenizer's count applies to another model.
• Forgetting that the reply needs room too.
• Expecting letter-level or digit-level precision from a model that sees neither.
• Treating token boundaries as meaningful. They follow frequency, not grammar.

Tokens and context window management are part of the Advanced LLM Concepts module of the Generative AI course.

Sources

• tiktoken: a fast BPE tokeniser for use with OpenAI's models (GitHub) — OpenAI
• Tokenization algorithms (Hugging Face Transformers documentation) — Hugging Face
• Neural Machine Translation of Rare Words with Subword Units (arXiv) — Sennrich, Haddow and Birch
• Key concepts (OpenAI API documentation) — OpenAI
• Understand and count tokens (Gemini API documentation) — Google
• Glossary (Claude Platform Docs) — Anthropic