Is this the exact same as OpenAI's tiktoken?

It implements the same regex pre-tokenization and byte-level pre-processing that cl100k_base and o200k_base use, which reproduces token boundaries and counts very closely. Because the full merge tables ship as megabytes of binary, the final count is a high-accuracy estimate rather than a byte-identical match to the server-side tokenizer.

What is the difference between cl100k_base and o200k_base?

cl100k_base is the encoding used by GPT-3.5-turbo and GPT-4, with a vocabulary of about 100k tokens. o200k_base is the larger ~200k-token vocabulary used by GPT-4o and the o-series, which packs more text per token — especially for code and non-English languages — so the same text usually costs slightly fewer tokens.

Why do whitespace and capitalization change the token count?

GPT tokenizers attach leading spaces to words, so " token" and "token" are different tokens. Capitalization and punctuation also split differently. This is why reformatting a prompt can quietly change its cost.

Are special tokens counted?

The tool flags special tokens like if it detects them in your text and notes that they are reserved control tokens. In real API calls the chat template adds its own role and delimiter tokens on top of your visible text.

Is my text sent anywhere?

No. All tokenization runs locally in your browser using JavaScript. Nothing you type is uploaded, logged, or stored.

Tiktoken Browser Tool

Tiktoken in the browser

Token count is the unit you pay in. This tool tokenizes text the way GPT models do — applying the regex pre-tokenization and byte-level encoding that cl100k_base and o200k_base use — so you can see how a prompt splits and roughly how many tokens it will cost before you spend a single API credit.

How it works

OpenAI’s BPE tokenizers run in two stages. First a regex splits the text into candidate pieces — words, leading-space-plus-word, numbers in groups, punctuation runs, and whitespace. Each piece is then encoded to UTF-8 bytes and merged with byte-pair encoding against the vocabulary. This tool reproduces stage one exactly (the official cl100k and o200k split patterns) and applies a faithful byte-pair merge heuristic to stage two, giving you token boundaries, the byte fragments inside each token, and a count that tracks the real tokenizer closely.

The two encodings differ mainly in vocabulary size. o200k_base has roughly twice the vocabulary, which lets it represent common multi-character sequences (and a lot of non-English text and code) as single tokens, so the same input typically yields fewer tokens than under cl100k_base.

Tips and notes

A leading space is part of the next token: ” hello” is usually one token, while “hello” at the start of a string may split differently. Watch this when concatenating strings.
Numbers are chunked in groups of up to three digits, so “1000000” is several tokens, not one.
Special control tokens such as <|endoftext|> are reserved — the tool flags them so you do not accidentally inject one. The visible text count here does not include the role and delimiter tokens the chat API adds around your message.
For exact billing always trust the usage field returned by the API; use this tool for fast local estimates while you iterate on a prompt.