What is a RAG evaluation triple?

A triple is one test case for a retrieval-augmented generation system. It pairs a question with the ground-truth context that should be retrieved to answer it, and the expected (reference) answer. Running your pipeline against many triples lets you measure retrieval quality and answer faithfulness objectively instead of by spot-checking.

What is the difference between the RAGAS and TruLens exports?

RAGAS expects columnar data — parallel arrays of questions, ground truths, and contexts — which maps to its Dataset.from_dict constructor. TruLens works with per-record objects. This tool produces both shapes from the same triples so you can switch harnesses without re-entering data. The custom format is a plain array of {question, contexts, answer} objects for your own scripts.

Should context be chunk IDs or full text?

Either works, and which you use depends on what your evaluator compares. If you score retrieval by exact chunk match, use stable chunk IDs. If you score by semantic overlap (e.g. context precision/recall metrics), use the actual chunk text. Enter one chunk per line; the tool splits them into a list automatically.

How many triples do I need for a useful eval set?

Start with 20-50 hand-written triples that cover your real query distribution, including edge cases and questions your system currently gets wrong. A small, high-quality, regression-style set you run on every change is far more valuable than a large noisy one.

Is my data sent anywhere?

No. The builder runs entirely in your browser and the JSON is assembled locally. You can paste proprietary documents and answers safely.

RAG Eval Dataset Builder

You can’t improve a RAG pipeline you don’t measure. The first thing every serious RAG project needs is an evaluation set — a list of questions paired with the context that should be retrieved and the answer the system should produce. This builder gives you a structured form for those triples and exports them in the exact shape your eval harness expects.

How it works

Each row is one test case with three parts:

Question — a real user query your system should handle.
Ground-truth context — the chunk(s) that contain the answer, one per line. Use stable chunk IDs if you score retrieval by exact match, or the chunk text if you score by semantic overlap.
Expected answer — the reference response a correct system would give.

Pick an export format and the tool assembles the JSON live. RAGAS output is columnar (parallel arrays) and drops straight into Dataset.from_dict. TruLens output is a list of records with query, expected_response, and expected_chunks. The custom format is a clean array of {question, contexts, answer} objects for your own scripts.

Why ground truth matters

Retrieval metrics like context precision and recall, and generation metrics like faithfulness and answer correctness, all need a reference to compare against. Without ground-truth context you can only measure whether the final answer looks right — you can’t tell whether a good answer came from good retrieval or from the model guessing. Capturing the expected chunks lets you separate retrieval failures from generation failures, which is usually where the real bug lives.

Tips

Seed the set with questions your system currently answers wrong — those are your highest-value regression tests.
Keep the set in version control next to your pipeline and re-run it on every prompt or index change.
Cover the full query distribution: easy lookups, multi-hop questions, and “no answer in the corpus” cases where the correct behaviour is to abstain.