What is a special token?

A special token is a reserved symbol the model treats as structure rather than literal text — markers like (begin a chat turn), (document boundary), BOS/EOS (sequence start/end), and [INST] (Llama instruction wrapper). They steer how the model interprets a sequence and are usually added automatically by the chat template.

Should I put special tokens in my own text?

Almost never. If you type into a user message, a model may interpret it as a real control token and break formatting or open an injection vector. Let the API's chat template insert them; this tool flags any you have accidentally included.

Why do different models use different tokens?

Each model family ships its own chat template. OpenAI models use ChatML ( role ... ), Llama uses [INST]...[/INST] and special header tokens, Mistral uses a similar instruction wrapper, and Gemini uses its own turn markers. Sending one family's tokens to another model does nothing useful.

Do special tokens cost money?

Yes — each one is a token you are billed for, and the chat template can add several per message (role markers, turn delimiters, BOS/EOS). For short, high-volume messages this overhead is a real fraction of cost, which the companion chat-template overhead analyzer quantifies.

Is my input uploaded?

No. Detection runs entirely in your browser with pattern matching. Nothing you paste is transmitted.

Special Token Decoder

Special token decoder

Special tokens are the invisible scaffolding of every LLM conversation — the <|im_start|> and [INST] markers that tell the model where a turn begins, who is speaking, and when the document ends. This tool scans your prompt or chat JSON, lists every special token it finds, explains what each one does, and warns you about any you have accidentally typed into user content.

How it works

The decoder matches your input against the known special-token vocabularies of major model families — ChatML (OpenAI), Llama, Mistral, and Gemini. For each match it reports:

the literal token,
its role (turn start, turn end, sequence boundary, instruction wrapper, etc.),
whether it is normally template-inserted (added by the API) or manual (typed by you), and
an estimated token cost of one each.

It also flags tokens that belong to a different family than the one you selected, which is a common copy-paste mistake when adapting a prompt across models.

Tips and notes

Treat special tokens as structure you should never type by hand. If you need a literal <|endoftext|> in user text, the safe pattern is to let the model never see it as a control token — most APIs reject or escape it for you, but do not rely on that.
Mixing families silently fails: [INST] does nothing on a GPT model, and <|im_start|> does nothing on raw Llama. The decoder highlights mismatches.
The total special-token count is your fixed per-message overhead before any visible content — pair this with the chat-template overhead analyzer to see the monthly cost of that scaffolding.
A user message containing what looks like a control token is a classic prompt-injection signal; flag and sanitize it server-side.