Does this run a real tokenizer?

No. It generates test scaffolding that asserts your own countTokens function returns the expected values. You wire it to your real tokenizer (tiktoken, the Anthropic SDK, etc.). The tool only writes the test boilerplate.

Why test token counts at all?

Token counts drive your billing estimates, context-window truncation, and rate-limit budgeting. A library upgrade or model swap can silently change counts, breaking cost forecasts and truncating prompts — a test catches that in CI.

How do I get the expected counts to put in?

Run your current tokenizer once against each sample and record the count as the expected value. The test then pins that behaviour so future changes are caught.

Is my sample data sent anywhere?

No. The generator runs entirely in your browser. Your sample texts and counts are never uploaded, stored, or logged.

Can I test multiple models?

Yes. Generate one test block per tokenizer and rename the function reference, or parameterize the test with a model column — the generated structure is easy to extend.

Token Counting Unit Test Generator

Pin your token counts with generated unit tests

Token counts are load-bearing: they set your cost estimates, decide when you truncate context, and gate your rate limits. When a tokenizer library upgrade or a model swap silently shifts those counts, your budgets drift and prompts get cut. This tool generates ready-to-run unit tests that assert your countTokens function returns the exact values you expect.

How it works

You provide sample texts and the token count each one should produce. The generator emits one assertion per case in your chosen framework:

expect(countTokens("Hello, world")).toBe(3);

The test calls your tokenizer wrapper, so it pins real behaviour. Wire countTokens to tiktoken, the Anthropic SDK’s counting endpoint, or whatever your integration uses, and the suite fails loudly the moment counts change.

Tips for a useful test suite

Cover edge cases. Include empty strings, emoji, code blocks, and non-Latin text — these are where tokenizers diverge most between versions.
Record counts from your current tokenizer. Run it once, capture the numbers, and pin them. The test guards against change, not against a theoretical “correct” answer.
Run it in CI. A token-count test is cheap and catches dependency drift before it reaches your billing dashboard or truncates a user’s prompt.