Llama and Mistral token counter
Self-hosting an open model or calling it through Groq, Together, or Fireworks means you pay and plan around tokens, not words. This tool estimates how many tokens a piece of text resolves to for Llama 2, Llama 3, and Mistral, so you can size prompts against a context window or project inference cost before you ship.
How it works
Llama and Mistral use SentencePiece/BPE tokenizers with different vocabulary sizes. Llama 2 and Mistral share a roughly 32,000-token vocabulary, while Llama 3 jumped to a 128,000-token vocabulary that packs more text into each token. Because a faithful in-browser tokenizer would require shipping the full merge tables, this tool uses calibrated characters-per-token ratios per family — about 3.6 chars/token for Llama 2 and Mistral and about 4.0 for Llama 3 — blended with a word-based estimate. That blend tracks the real tokenizer within roughly 5-10% on ordinary English.
Tips and notes
Token counts vary by language and content type: non-Latin scripts and emoji cost far more tokens per character, while repetitive or boilerplate text costs fewer. Reserve headroom in your context budget — a “4K” model can rarely fit a literal 4,096-token prompt once the response and special tokens are accounted for. For billing-critical decisions, confirm with the model’s own tokenizer.