What is an LLM-as-judge?

It is using one model to score another model's output against a rubric. A clear, weighted rubric makes those judgments consistent and lets you compare prompts, models, or versions on the same scale.

Why are criteria weighted?

Without weights, a great-looking but factually wrong answer could outscore a plain but correct one. Weighting accuracy and safety higher than formatting keeps the score aligned with what really matters.

Should I use the same rubric for every task?

Keep the shared core criteria for comparability, but add a task-specific criterion. Summarization needs faithfulness, code needs runnability, RAG needs grounding — a one-size rubric misses these.

How do the scoring bands work?

The rubric suggests pass at 85% or above, revise between 60 and 85, and reject below 60 of the weighted maximum. Tune these thresholds to your risk tolerance and the cost of a bad output.

Can I use this for human reviewers too?

Yes. The 0–3 descriptors give human graders concrete anchors, which dramatically improves inter-rater agreement compared to vague "rate 1–10" instructions.

AI Output Evaluation Rubric Generator

AI output evaluation rubric generator

“Is this AI output good?” is unanswerable until you define good. This tool turns that vague question into a concrete, weighted scoring rubric you can apply consistently — whether a human grader or another model does the scoring. It always includes core criteria (accuracy, helpfulness, safety, instruction following) and adds a tailored dimension for your specific task type.

How it works

You select a task type — summarization, classification, code generation, creative writing, RAG, structured extraction, or general — and the generator prepends a domain-specific criterion with its own 0–3 descriptors (faithfulness for summaries, runnability for code, grounding for RAG, and so on). You can name the use case and add extra dimensions like brand tone or empathy. Each criterion carries a weight, so accuracy and safety count more than formatting, and the tool computes the weighted maximum and suggested pass/revise/reject bands. The output is clean Markdown you can paste into a review doc or use as LLM-as-judge instructions.

Tips and notes

Anchor every level. The 0–3 descriptors are what make scores reproducible; vague “rate it 1–10” instructions produce noise.
Keep the core, swap the domain criterion. Sharing core criteria lets you compare across tasks while the domain criterion captures what’s unique.
Don’t let style outvote correctness. The default weights put accuracy and safety at ×3 for exactly this reason — preserve that when editing.
Calibrate the bands. A medical or legal use case warrants stricter thresholds than an internal brainstorming tool.