Retrieval Evaluation: Measuring RAG Quality

A RAG system that gives a wrong answer has failed in one of two places. Either the right passage was never retrieved, or it was retrieved and the model misused it. The fixes are completely different, and without measurement you cannot tell which you are facing.

Retrieval evaluation is the cheaper half to measure, and the more useful. It needs no language model and no judgement calls: for each test question, was the passage that answers it among the results, and how high?

Build the test set first

Everything depends on a set of questions paired with the passages that answer them.

1. Collect real questions. Support tickets, search logs and questions colleagues actually ask beat anything invented. If you must write them, do not copy phrases from the documents, or you will only test word matching.
2. Label the answer. For each question, record which document or chunk contains it.
3. Include hard cases: questions in users' vocabulary instead of the documents', questions with exact codes or names, and a few that the documents cannot answer.
4. Keep it fixed. The same questions must be run before and after every change.

Thirty to fifty questions are enough to start. As shown below, they are not enough to settle close contests.

The metrics

All of these are computed from one fact per question: the rank at which the right passage appeared.

Hit rate is the one to start with, because it matches how RAG works. If you send five passages to the model, hit rate@5 is the share of questions for which the model had the answer in front of it at all. Recall, in this sense, is about not missing what matters (Ragas: context recall).

MRR and nDCG reward putting the answer higher. nDCG discounts each rank logarithmically and divides by the score of a perfect ranking (scikit-learn: ndcg_score). It also handles several relevant passages with different grades of relevance, which MRR does not.

Precision matters when irrelevant passages do harm, by distracting the model or by costing tokens. Ragas builds its context precision measure on precision at each rank, to reward retrievers that put relevant chunks above irrelevant ones (Ragas: context precision).

Computing them

The script evaluates two retrievers on the same 40 questions over 148 sections of public-domain Python documents (peps.python.org): one using embeddings alone, one a hybrid of embeddings and keyword search. Each question has one section that answers it.

import json
import re
from math import comb, log2

import numpy as np
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer

sections = json.load(open("../../genai/pep_sections.json", encoding="utf-8"))
questions = json.load(open("../../genai/pep_questions.json", encoding="utf-8"))
ids = [s["id"] for s in sections]
texts = [s["title"] + "\n" + s["text"] for s in sections]
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
vectors = model.encode(texts, normalize_embeddings=True)
tokens = lambda text: re.findall(r"[a-z0-9_]+", text.lower())
bm25 = BM25Okapi([tokens(t) for t in texts])


def rank_of_answer(question, target, hybrid):
    """Position of the section that answers the question in the ranked results, starting at 1."""
    dense = np.argsort(-(vectors @ model.encode(question, normalize_embeddings=True)), kind="stable")
    if not hybrid:
        return list(dense).index(target) + 1
    keyword = np.argsort(-bm25.get_scores(tokens(question)), kind="stable")
    fused = np.zeros(len(texts))
    for ranking in (dense, keyword):
        fused[ranking] += 1 / (60 + np.arange(1, len(texts) + 1))
    return list(np.argsort(-fused, kind="stable")).index(target) + 1


targets = [ids.index(q["answer_in"]) for q in questions]
dense = np.array([rank_of_answer(q["question"], t, False) for q, t in zip(questions, targets)])
hybrid = np.array([rank_of_answer(q["question"], t, True) for q, t in zip(questions, targets)])


def metrics(ranks):
    return {"hit@1": np.mean(ranks <= 1), "hit@5": np.mean(ranks <= 5), "hit@10": np.mean(ranks <= 10),
            "MRR": np.mean(1 / ranks), "nDCG@10": np.mean([1 / log2(r + 1) if r <= 10 else 0 for r in ranks])}


print(f"{len(questions)} questions, one answering section each, {len(sections)} sections to search")
print("retriever    " + "".join(f"{name:>9}" for name in metrics(dense)))
for name, ranks in (("embeddings", dense), ("hybrid", hybrid)):
    print(f"{name:12} " + "".join(f"{value:>9.3f}" for value in metrics(ranks).values()))

rng = np.random.default_rng(0)                     # how sure can 40 questions make us?
resampled = [np.mean(rng.choice(dense, len(dense)) <= 5) for _ in range(10_000)]
low, high = np.percentile(resampled, [2.5, 97.5])
print()
print(f"hit@5 for embeddings: {np.mean(dense <= 5):.3f}, 95% interval from resampling {low:.3f} to {high:.3f}")

better, worse = int(np.sum(hybrid < dense)), int(np.sum(hybrid > dense))
changed = better + worse
p = min(1.0, 2 * sum(comb(changed, k) for k in range(max(better, worse), changed + 1)) / 2 ** changed)
print(f"hybrid against embeddings, question by question: better on {better}, worse on {worse}, "
      f"same on {len(dense) - changed}; sign test p = {p:.2f}")

print()
print("questions the embeddings retriever missed (answer outside the top 10):")
for q, r in zip(questions, dense):
    if r > 10:
        print(f"  rank {r:>3}  {q['question']}")

40 questions, one answering section each, 148 sections to search
retriever        hit@1    hit@5   hit@10      MRR  nDCG@10
embeddings       0.525    0.825    0.850    0.650    0.693
hybrid           0.525    0.900    0.900    0.685    0.735

hit@5 for embeddings: 0.825, 95% interval from resampling 0.700 to 0.925
hybrid against embeddings, question by question: better on 15, worse on 8, same on 17; sign test p = 0.21

questions the embeddings retriever missed (answer outside the top 10):
  rank  18  How should I check whether a variable is None?
  rank  39  Is it fine to assign a lambda expression to a variable?
  rank  15  How can I give a complicated type a shorter name?
  rank  16  How do I say that a value can be one of several types?
  rank  12  How can I refer to a class in an annotation before the class is defined?
  rank  25  What symbol is used for an assignment expression?

Reading the table

With embeddings alone, the right section was first for 52.5% of questions and in the top five for 82.5%. The hybrid has the same share at first place and 90% in the top five. Its MRR and nDCG are higher too. On the face of it, the hybrid is better.

How sure is that

The next two lines of output are the ones that matter.

The interval. Resampling the 40 questions gives a 95% interval for the embeddings' hit rate@5 of 0.700 to 0.925. The measured 0.825 could plausibly have been anywhere in that range with a different set of 40 questions. The hybrid's 0.900 lies inside it.

The paired comparison. Question by question, the hybrid ranked the answer higher 15 times and lower 8 times. A sign test on those 23 changes gives p = 0.21. That is weak evidence. A difference this size would appear by chance about one time in five if the two retrievers were equally good.

So the honest conclusion is: the hybrid is probably better, and this test set cannot prove it. To separate two systems that differ by a few points you need a few hundred questions, not forty. This matters in practice. Teams change a chunk size, see hit rate move from 0.82 to 0.85 on 40 questions and ship it. That movement is noise.

Reading the failures

The last part of the output is where the improvements come from. Six questions were missed, and they are not random:

• Two are about rules inside one very long section. That is a chunking problem. See chunking strategies for RAG.
• Three use different words from the documents: "a shorter name" for an alias, "one of several types" for a union, "before the class is defined" for a forward reference. That is a vocabulary gap.
• One asks for a symbol, which an embedding of the question cannot match.

Each group suggests a different fix. A single number would have suggested none.

Beyond retrieval

Good retrieval is necessary and not sufficient. The answer the model writes needs its own checks:

• Faithfulness. Is every claim in the answer supported by the retrieved passages? Ragas defines it as the supported claims divided by all the claims in the response (Ragas: faithfulness).
• Correctness. Does the answer match a reference answer?
• Refusal. When the documents do not contain the answer, does the system say so?

These usually need a language model as a judge, or human review, and are covered in evaluating LLM outputs.

Measure retrieval first. If the passage was not retrieved, no judgement of the answer is needed to know what went wrong.

A warning about benchmarks

Public leaderboards rank embedding models and retrievers, and it is tempting to pick the top one. A large study across 18 retrieval datasets found that plain keyword search with BM25 was a robust baseline, and that dense retrievers often did worse than other methods on data unlike what they were trained on (Thakur et al.: BEIR). A model's rank on someone else's data predicts its rank on yours only loosely. Your own test set is the benchmark that counts. See semantic search vs keyword search.

Common mistakes

• No test set, only a feeling that it works.
• Writing questions by paraphrasing the documents, which flatters every retriever.
• Declaring a winner on a small difference.
• Reporting one number and not reading the failures.
• Evaluating answers before retrieval.
• Changing the test set between runs.
• Testing only questions that have answers.

For the pipeline being measured, see RAG architecture explained.

LLM evaluation and prompt evaluation are part of the Evaluation and Optimization module of the Generative AI course.

Sources

• BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models (arXiv) — Thakur et al.
• ndcg_score (scikit-learn documentation) — scikit-learn
• Context Precision (Ragas documentation) — Ragas
• Context Recall (Ragas documentation) — Ragas
• Faithfulness (Ragas documentation) — Ragas
• Python Enhancement Proposals (Python Software Foundation) — Python Software Foundation