Why build a benchmark at all?

Without a fixed test set you cannot tell whether a prompt change or model upgrade improved or regressed quality. A benchmark turns "it feels better" into a measurable score you can track over time.

How many test cases do I need?

Twenty is a reasonable floor for a single task; the tool nudges you toward it. Include normal cases, edge cases, and adversarial inputs so the benchmark exercises the prompt's weak spots, not just the happy path.

What scoring types are supported?

Exact match for deterministic outputs like labels, semantic or fuzzy match for free-text answers, and rubric scoring where you define point-based criteria for a grader (human or LLM) to apply.

What format is the export?

A clean JSON object with the task, scoring type, optional rubric, a version number, and a numbered array of input and expected-output cases. It loads directly into most eval frameworks.

Does the tool run the evaluation?

No. It builds the benchmark file. You replay it against your model in your own harness, which keeps the test data and the model under your control.

Prompt Benchmark Builder

Prompt benchmark builder

You cannot improve what you do not measure. Tweaking a prompt by feel works until a model upgrade silently breaks half your cases and you have no way to notice. A benchmark fixes a set of inputs with known-good outputs and a scoring rule, so every prompt or model change produces a comparable score. This builder helps you assemble that benchmark and export it as portable JSON.

How it works

You describe the task and pick a scoring type: exact match for labels and structured output, semantic match for free text, or a rubric where you write point-based criteria. Then you add input and expected-output pairs through the table, aiming for at least twenty that cover normal, edge, and adversarial cases. The tool renumbers them, wraps everything in a versioned JSON object with the task and scoring rule, and lets you copy it. Drop that file into your own eval harness to replay the benchmark against any model or prompt revision.

Tips and notes

Cover the hard cases. A benchmark of easy inputs always passes and tells you nothing. Include the ambiguous and adversarial ones.
Version your benchmark. The export carries a version field — bump it when you change the cases so historical scores stay comparable.
Match scoring to output. Use exact match for labels and JSON; reach for a rubric only when correctness is genuinely subjective.
Re-run on every change. Replay the benchmark after each prompt edit and each model upgrade — that is the entire point of having one.