What is a token?
A token is the basic unit of text an AI language model reads and writes. It is usually a short chunk of characters — often a whole common word, a word fragment, a punctuation mark, or a space. Models like GPT, Claude and Gemini never see “words” the way you do; they convert your text into a sequence of tokens, predict the next token over and over, and convert the tokens back into text.
How tokenization works
A tokenizer splits text using a fixed vocabulary learned during training (commonly via an algorithm called byte-pair encoding). Frequent words become a single token; rarer or longer words split into several. For example:
cat→ 1 tokentokenization→ often 2–3 tokens🙂or an unusual symbol → 1 or more tokens
Because the vocabulary is fixed, the same tokenizer can handle any language, source code, or text it has never seen before.
Why tokens matter
Tokens are the unit of measurement, cost and limits for every modern AI API:
- Cost — you are billed per token, with input and output priced separately.
- Context window — each model can only hold so many tokens at once (your prompt and its answer must fit). Larger windows hold more documents or conversation history.
- Speed — longer token sequences take longer and cost more to process.
How to estimate tokens from words
A quick rule of thumb for English:
- 1 word ≈ 1.3 tokens
- 100 tokens ≈ 75 words
- 1,000 tokens ≈ 750 words ≈ 1.5 pages
For an exact count, use a model-specific token counter, and to estimate spend try the LLM API Cost Calculator.