Is this the exact GPT-4o tokenizer?

It is a close approximation of cl100k_base's byte-pair behaviour, not the exact vocabulary. It reproduces the visible rules — leading-space attachment, splitting on case and digits, and subword breaks — so counts are usually within a few percent of the real tokenizer for English.

Why does a leading space belong to the next word?

GPT tokenizers attach a leading space to the token that follows it, so ' token' is one token, not a space plus a word. The visualizer shows this by including the space inside the colored chip.

Why do numbers split into single digits?

cl100k_base often splits long digit runs into chunks, so 2026 may become two or more tokens. The visualizer mirrors this so you can see why numeric-heavy text costs more tokens than you would expect.

Does it handle other languages?

It handles them, but non-English and CJK text often uses many more tokens per character because the byte-pair vocabulary is English-biased. The colored chips make that overhead obvious.

No. Tokenization runs entirely in your browser. Nothing you paste leaves the page.

Token Visualizer — Gera Tools

Token visualizer

Models do not read characters or words — they read tokens. Seeing exactly where your text breaks into tokens explains why billing, context limits, and even model behaviour sometimes surprise you. This visualizer colors each token as its own chip so you can spot subword splits, whitespace handling, and the hidden cost of numbers and rare words.

How it works

The tool applies a byte-pair-style splitter modeled on cl100k_base, the tokenizer GPT-4o uses. It first segments on the same boundaries the real regex uses — contractions, runs of letters, runs of digits, punctuation, and leading spaces — then merges short common fragments the way BPE would. Crucially, a leading space stays attached to the word after it, which is why token counts as one token rather than two. Long digit runs and uncommon words break into multiple subword tokens, which the colored chips make immediately visible.

Tips and notes

If a chip splits a common word into pieces, that word is rare in the vocabulary — rephrasing with more common words can shave tokens off long prompts.
Numbers, IDs, and hashes are token-hungry; consider whether you really need them verbatim in a prompt you send thousands of times.
For an exact production count, call the provider’s tokenizer or count_tokens endpoint — use this visualizer for intuition and quick estimates.