RAG retrieval evaluation: calculate recall, precision and reciprocal rank

Evaluate a RAG retriever by checking whether it returns the evidence a query needs, before asking whether a language model produces a good answer. Retrieval recall, precision and rank-based measures answer different questions about that evidence. A fluent answer can hide a retrieval miss, while a successful retrieval can still be followed by an unsupported answer.

We will create a tiny offline evaluation set with invented document IDs and explicit relevance judgments. Two sets of rankings stand in for two retriever configurations. No model API, paid service or downloaded document collection is required: the example isolates the evaluation arithmetic.

Define relevance before scoring

Suppose a documentation collection contains pages about setup, exports, sharing and account settings. For each query, a reviewer identifies which document IDs provide the required evidence. These are our relevance judgments, not a metric inferred from a model's confidence.

In the script, setup requires document d1; export can use d2 and d3; sharing requires d4. Here both export documents count as relevant evidence. In a real task, clarify whether documents are alternatives or jointly necessary: finding one of several relevant documents is not always enough to answer completely.

relevant = {"setup": {"d1"}, "export": {"d2", "d3"}, "sharing": {"d4"}}
rankings = {
    "A": {"setup": ["d5", "d1"], "export": ["d2", "d5"], "sharing": ["d5", "d6"]},
    "B": {"setup": ["d1", "d5"], "export": ["d2", "d3"], "sharing": ["d4", "d5"]},
}

def score(retrieved, expected, k=2):
    top = retrieved[:k]
    if not expected or len(top) != k or len(set(top)) != len(top):
        raise ValueError("this example requires judged queries and k unique results")
    hits = len(set(top) & expected)
    reciprocal_rank = next((1 / rank for rank, doc in enumerate(top, 1)
                            if doc in expected), 0.0)
    return hits / k, hits / len(expected), reciprocal_rank

for name, results in rankings.items():
    values = [score(results[query], expected) for query, expected in relevant.items()]
    averages = [sum(row[column] for row in values) / len(values)
                for column in range(3)]
    print(name, "precision@2=%.3f recall@2=%.3f MRR@2=%.3f" % tuple(averages))

assert score(["d2", "d5"], {"d2", "d3"}) == (0.5, 0.5, 1.0)

Expected output:

A precision@2=0.333 recall@2=0.500 MRR@2=0.500
B precision@2=0.667 recall@2=1.000 MRR@2=1.000

These are unweighted averages over three queries. B performs better on these invented judgments; the tiny example provides no evidence that a real retrieval method will outperform another in deployment.

Read the measures separately

Precision at two is the fraction of the two retrieved documents that are relevant. Recall at two is the fraction of the judged relevant documents retrieved. Reciprocal rank is one divided by the rank of the first relevant result, or zero when none is found within the cutoff. Averaging that last measure gives the truncated mean reciprocal rank reported here.

The export query illustrates their differences. Configuration A retrieves one relevant document first, so reciprocal rank is 1. Yet recall is only 0.5 because the second relevant document is absent. A strong first-hit measure does not establish complete evidence coverage.

Larger retrieval evaluations may also use graded relevance and measures such as NDCG. Choose a measure that reflects whether rank, coverage or degrees of usefulness matter to the application. The ir_measures reference documents standard retrieval-measure definitions and cutoff conventions.

Build a useful evaluation set

Include realistic questions, paraphrases, ambiguous requests and questions the collection cannot answer. Record the corpus version, relevance judgments, chunking configuration and retrieval cutoff. Keep tuning queries separate from the final evaluation set so repeated optimization does not quietly turn the test into training data.

Judgments are often incomplete: an unlabelled document may be useful rather than irrelevant. Review apparent false positives and newly retrieved evidence. If chunks overlap heavily, define whether relevance is scored at chunk, section or document level and prevent duplicate evidence from inflating the result.

Our function rejects duplicate document IDs and unjudged queries to keep the toy arithmetic explicit. A production evaluator needs a documented policy for fewer than k results, empty relevance sets and failures. Do not silently drop difficult queries; report the denominator and failure rate.

Separate retrieval from answer evaluation

After retrieval, test whether the generated answer is correct, supported by the retrieved evidence, appropriately cited and willing to abstain when evidence is missing. Keep those measurements separate so a failure can be traced to retrieval, context construction or generation.

Increasing k may improve recall while adding irrelevant context, latency and cost. A retrieval metric alone cannot select the best overall system. Run controlled comparisons on the actual task and inspect representative errors.

As an exercise, move a relevant document from rank one to rank two. Which measure changes? Then remove one export document from the relevance judgments and explain how that annotation decision changes recall. Continue with structured-output validation and Generative AI resources to test the next boundary in an LLM application.