Why does few-shot diversity matter?

Models generalize from the examples you give them, so if every example uses the same structure or vocabulary the model will overfit to that pattern and handle off-distribution inputs poorly. Diverse examples teach the model the task rather than one narrow phrasing of it.

How is similarity measured?

Each pair of examples is compared with Jaccard similarity over their word sets — the size of the shared vocabulary divided by the combined vocabulary. A score near 1 means the two examples use almost identical words; near 0 means they barely overlap. Pairs above the threshold are flagged.

What is a good diversity score?

There is no single right number, but you generally want low pairwise similarity (most pairs well under 0.5) and meaningful length variation. If two examples are nearly identical, one of them is wasting a slot in your prompt and may bias the model.

Does length variation matter too?

Yes. If all your examples are the same length the model may learn to always produce outputs of that length. Showing short and long examples teaches it to match the input rather than a fixed template, so the tool reports the length spread as well.

Few-Shot Example Diversity Checker

Make sure your prompt examples actually teach the task

Few-shot prompting works because the model imitates the examples you provide. The failure mode is subtle: if your examples are all phrased the same way, the model learns the phrasing instead of the task and breaks on inputs that look different. This checker measures how varied your examples really are — across vocabulary, length, and pairwise overlap — so you can catch redundancy before it biases your prompt.

How it works

Each example is tokenized into a set of words. The tool computes three things. First, lexical diversity: the ratio of unique words to total words across all examples, which tells you how much vocabulary your set covers. Second, length variation: the spread between your shortest and longest example, since uniform lengths push the model toward fixed-length outputs. Third, and most usefully, pairwise Jaccard similarity — for every pair of examples it divides the shared vocabulary by the combined vocabulary and flags any pair above a similarity threshold as too alike.

Tips and notes

Aim for examples that differ in structure, length, and wording while still demonstrating the same task. Two near-identical examples waste a slot and can nudge the model toward that exact pattern, so when a pair is flagged, replace one of them with a genuinely different case — ideally an edge case or a different input shape. Diversity is not the only goal: every example should still be correct and representative. Use this tool as a redundancy filter, not a correctness check. You can paste either plain blocks separated by blank lines or a JSON array of strings; both are parsed automatically.