What is recall@k versus precision@k?

Recall@k is the fraction of all relevant chunks that appear in the top k results — it answers "did we retrieve enough of the right context?" Precision@k is the fraction of the top k that are actually relevant — it answers "how much noise are we paying tokens for?" Raising k usually raises recall but lowers precision, so the metrics trade off against each other.

What is MRR (Mean Reciprocal Rank)?

Reciprocal rank is 1 divided by the position of the first relevant result — so a relevant chunk at rank 1 scores 1.0, at rank 2 scores 0.5, at rank 3 scores 0.33, and so on. Averaging the reciprocal rank across all your test queries gives MRR. It rewards systems that put a relevant result near the very top, which matters when you only feed the LLM a few chunks.

How do I choose the right k for my RAG app?

Pick the smallest k where recall@k is high enough that the answer is almost always present, then stop — every extra chunk costs context tokens and can dilute the model with irrelevant text. This simulator lets you see exactly where recall plateaus so you don't overpay for k.

Where do the similarity scores come from?

You enter them manually — typically the cosine similarity or distance your vector database returns for each chunk against the query. The simulator only needs the relative ordering, so any consistent score works. This lets you model real or hypothetical retrieval results without standing up a database.

Is my data sent anywhere?

No. Ranking and all metric calculations run locally in your browser. You can paste real chunk text and scores safely.

Top-K Retrieval Simulator

Tuning a RAG retriever comes down to one question: how many chunks should I return? Too few and the answer isn’t in the context; too many and you waste tokens and confuse the model. This simulator lets you play out the trade-off with your own scores before touching the pipeline.

How it works

Enter the query, then each candidate chunk with the similarity score your vector search assigned and a checkbox marking whether it is genuinely relevant (your ground truth). The tool sorts the chunks by score — exactly as a real top-k retriever would — and then, for k = 1, 3, 5, and 10, computes:

Hit@k — was at least one relevant chunk returned?
Recall@k — what fraction of all relevant chunks made it into the top k?
Precision@k — what fraction of the top k were relevant?

It also reports the reciprocal rank for this query: 1 divided by the position of the first relevant chunk. Average that across your whole test set and you have MRR.

Choosing k

Watch where recall@k stops climbing. If recall hits 100% at k = 3, returning 10 chunks just adds noise and token cost for no benefit. If recall is still low at k = 10, the problem isn’t k — it’s your embeddings or chunking, and no amount of widening the window will fix it. Precision@k tells you the opposite story: a low precision means the model is wading through irrelevant text to find the answer.

Tips

Run several representative queries and average the metrics; a single query can be misleading.
A low MRR with high recall means relevant chunks are retrieved but ranked poorly — a re-ranker is the usual fix.
Pair this with the RAG Eval Dataset Builder to turn ad-hoc tuning into a repeatable regression test.