How are tokens estimated?

The tool uses the common English heuristic of roughly four characters per token. It is an estimate, not an exact tokenizer count, so leave a little headroom below your model's real limit.

Why use overlap between chunks?

Overlap repeats the tail of one chunk at the start of the next so a sentence or idea that straddles a boundary still appears whole in at least one chunk. This improves retrieval recall in RAG pipelines.

Which boundary strategy should I pick?

Sentence-aware is the safest default for prose. Paragraph-aware keeps logical sections together for structured docs. Hard character splitting is only for data with no natural boundaries.

Does chunking happen in my browser?

Yes. All splitting runs locally in JavaScript and nothing is uploaded. Your document never leaves the page.

What chunk size should I use?

For RAG, 200 to 500 tokens with 10 to 20 percent overlap is a common sweet spot. Smaller chunks improve precision; larger chunks preserve more context per retrieval. Tune against your own eval set.

Long Context Chunker

Long context chunker

When a document is larger than a model’s context window, or larger than a sensible retrieval unit, it has to be split. Cutting blindly at a character count slices sentences in half and degrades both embeddings and prompt quality. This chunker splits on sentence or paragraph boundaries, packs units up to your target chunk size, and carries an overlap between adjacent chunks so context that straddles a boundary survives. Everything runs in your browser.

How it works

You set a target chunk size in tokens and an overlap in tokens. The tool splits the text into units (sentences or paragraphs), then greedily fills each chunk until adding the next unit would exceed the target. When a chunk closes, the last few units are repeated at the start of the next chunk to provide overlap. Token counts are estimated at roughly four characters per token, so they are approximate — keep some headroom below your model’s hard limit. A hard character-split mode is available for data with no natural boundaries.

Tips and notes

Default to sentence-aware. It rarely breaks meaning and works for most prose and documentation.
Use 10 to 20 percent overlap. Enough to bridge boundaries without ballooning your token spend on duplicated text.
Estimates are approximate. Four-chars-per-token is a heuristic; verify against your model’s real tokenizer for tight budgets.
Smaller chunks, sharper retrieval. If RAG answers feel vague, shrink the chunk size before reaching for a bigger model.