A useful RAG evaluation separates two questions: did retrieval find evidence that can support the answer, and did the model use that evidence correctly? Start with versioned queries, corpus snapshots, and relevance judgments—not an uncalibrated LLM judge. Freeze chunking and index configuration, choose Recall@k, MRR, or nDCG@k according to the retrieval job, and record latency, context tokens, and failure slices. Offline regression, human review, and online evidence must agree before a candidate replaces the baseline.

Why is final-answer scoring insufficient?

A wrong answer can come from missed retrieval, poor ranking, context truncation, a prompt, model reasoning, or citation generation. A single answer score cannot tell the team whether to change embeddings, hybrid fusion, reranking, or generation. The inverse is also true: a high retrieval score does not guarantee a grounded answer. A model can ignore excellent evidence or combine it into a claim the sources do not support.

Use at least two layers:

  1. Retrieval evaluation asks whether relevant evidence enters the ranked candidates.
  2. Generation evaluation asks whether the answer uses the supplied context completely and faithfully.

This guide focuses on retrieval. If you are building lexical-plus-vector retrieval, the MongoDB $rankFusion hybrid-search guide provides an implementation baseline. The evaluation harness should treat lexical, vector, fusion, candidate count, and reranker settings as experiment variables rather than assuming one architecture wins.

What belongs in a reproducible test case?

A minimal case is not only a question and reference answer. It binds the information need to a corpus version and relevance judgments:

{
  "queryId": "q-0042",
  "query": "How should a publish retry behave after commit but cache failure?",
  "corpusVersion": "docs-2026-08-01",
  "relevant": [
    {"documentId": "publish-recovery", "grade": 3},
    {"documentId": "cas-state-machine", "grade": 2}
  ],
  "slice": ["recovery", "long-tail"],
  "notes": "Must preserve the original revision tuple"
}

Document or chunk IDs must be stable within a corpus version. Random IDs after every re-chunk make historical scores uninterpretable. Define the relevance unit explicitly. A relevant parent document does not mean every chunk can support the answer. For RAG, label the smallest passage that carries necessary evidence and retain the parent ID for analysis.

The NIST TREC relevance-judgment guidance illustrates the classic relationship among topics, documents, and qrels, and warns that the document collection must match the judgments. A product benchmark can be much smaller while preserving that version-consistency rule.

Every run should emit an immutable artifact containing code revision, corpus and index versions, preprocessing configuration, retrieval parameters, per-query rankings, metrics, latency, and failures. A dashboard average without its inputs is not a reproducible result.

Where should queries come from?

Prefer real information needs: authorized and redacted search logs, support questions, failed navigation, incident reviews, and product acceptance cases. Do not send raw user content to an external model merely to expand a dataset. Define authorization, redaction, retention, and access first.

A healthy set contains slices for frequent and long-tail needs, short and constrained questions, exact terminology and paraphrases, single- and multi-document evidence, time-sensitive facts, negation, permissions, and unanswerable requests. Questions generated only from headings are often lexically easy and inflate retrieval scores.

Separate a development set from a frozen release set. Tune on development data and run the release gate on the frozen set. Repeatedly selecting parameters against the release set turns it into training data. Add newly discovered failures to a new benchmark version while retaining previous versions; never improve a score by silently removing difficult cases.

Include a baseline that is cheap and understandable. BM25 or the current production retriever often serves this role. A candidate that beats a weak or misconfigured baseline provides little evidence. Preserve the exact baseline configuration just as carefully as the candidate.

How should relevance be judged?

Binary judgments answer whether a result carries usable evidence. Graded judgments distinguish direct support from partial or background relevance. Write a rubric before labeling:

  • 3: directly contains the key fact and its boundary conditions;
  • 2: contains necessary evidence that must be combined with another passage;
  • 1: topically related but insufficient to support the answer;
  • 0: irrelevant or misleading.

Have assessors label independently, then adjudicate disagreements. Measure disagreement by slice; systematic ambiguity may reveal a vague query or a chunking problem. Keep notes for high-impact cases so a future benchmark maintainer can understand the intended judgment.

Do not build the judgment pool only from the current retriever’s top results. A new system may find relevant evidence the old system never exposed, which would appear unjudged. Pool candidates from lexical, vector, hybrid, several parameter settings, rerankers, and targeted human search.

The original BEIR paper evaluates lexical, sparse, dense, late-interaction, and reranking approaches across heterogeneous domains. Its broader lesson for product teams is that a narrow average does not prove generalization. Report slices by document type, language, query intent, and update frequency.

What do Recall@k, MRR, and nDCG@k measure?

Recall@k: did the candidate set cover the evidence?

Recall@k is the fraction of all relevant items found in the first k results. It is a natural first-stage metric because candidate generation should avoid losing evidence before a reranker sees it. When each query has one required passage, teams sometimes report Hit@k. Define it explicitly: Hit@k asks whether at least one relevant item appeared and can hide failure on multi-evidence questions.

MRR: how early did the first usable result appear?

Mean reciprocal rank takes the reciprocal of the first relevant rank for each query, then averages across queries. It fits cases where one direct result is sufficient. It ignores every relevant result after the first, so it is a poor coverage metric for questions that require several pieces of evidence.

nDCG@k: were graded results ordered well?

Normalized discounted cumulative gain assigns gain according to relevance grade, discounts lower ranks, and normalizes against the ideal ordering. The Stanford Introduction to Information Retrieval chapter provides the formal definition. nDCG fits 0–3 judgments and rewards putting directly supporting passages above background material.

There is no universally best metric. A common design uses Recall@20 or Recall@50 for candidate generation, nDCG@5 for reranking, and MRR for direct-answer slices. Choose k from the real context budget. If generation receives at most eight chunks, Recall@100 can count evidence the product never uses.

What does a framework-independent evaluation loop look like?

def recall_at_k(ranked_ids, relevant_ids, k):
    expected = set(relevant_ids)
    if not expected:
        return None
    found = set(ranked_ids[:k]) & expected
    return len(found) / len(expected)

def reciprocal_rank(ranked_ids, relevant_ids):
    expected = set(relevant_ids)
    for rank, item_id in enumerate(ranked_ids, start=1):
        if item_id in expected:
            return 1.0 / rank
    return 0.0

Returning None for a query with no relevant items prevents the evaluator from silently treating it as zero or one. Unanswerable queries need a separate contract: can retrieval express insufficient confidence, and can generation abstain rather than fabricate?

For graded nDCG, use a tested library or compare a local implementation against known examples. Define how ties, duplicate IDs, missing judgments, and fewer than k results are handled. Version that definition. Two tools can use the same metric name while differing on edge cases.

If the harness calls a service API, give the task a strict input schema, explicit run state, and immutable result artifact. The FastAPI and MongoDB bilingual-publishing guide demonstrates strict models and state boundaries that transfer well to evaluation jobs. Avoid notebooks that overwrite the previous ranking output.

Why does re-chunking invalidate comparisons?

Changing from 500-token to 800-token chunks can invalidate IDs and judgments. A larger chunk may technically contain the answer while adding enough irrelevant text to hurt generation. Report both chunk-level and parent-document-level metrics, and migrate or relabel judgments when segmentation changes.

Text-overlap mapping can propose a new chunk for an old label, but it is not final evidence. Human-review the frozen release set. Record chunk size, overlap, heading inheritance, table conversion, code-block handling, and metadata filters. These preprocessing changes can move retrieval quality more than the embedding model.

Deduplication belongs in evaluation too. If top five contains five overlapping chunks from one parent, raw Recall may look acceptable while context diversity is poor. Report unique parent documents, overlap ratio, and evidence coverage after the exact deduplication step used by generation.

How do latency, cost, and context budget enter the gate?

Accuracy is one constraint. Record time spent in embedding, lexical retrieval, vector retrieval, fusion, reranking, and document fetch. Also record candidate count, final chunk count, deduplicated context tokens, cache hits, and external calls.

Configuration Recall@20 nDCG@5 p95 latency Context tokens Notes
baseline measured measured measured measured fixed production input
candidate measured measured measured measured proposed change

Those cells must contain real measurements. Until a run exists, label them as placeholders rather than inventing a percentage improvement. A release policy can require no overall regression, minimum results for critical slices, and latency and token budgets. Always list the query IDs with the largest gains and losses; averages can hide a severe permission or safety regression.

For multi-stage systems, measure recall before and after each stage. If vector search finds the passage but fusion removes it, improve fusion. If fusion preserves it and the reranker removes it, changing embeddings is unlikely to solve the measured failure.

What can an LLM judge do safely?

An LLM can propose queries, summarize ranking differences, and pre-label candidates. It should not be the uncalibrated sole source of truth. Judges can favor their own phrasing, change with model versions, miss permission boundaries, and follow instructions embedded in retrieved text.

If using one, pin model and prompt, hide system identity, require structured reasons with evidence spans, compare against a human-labeled calibration set, route uncertainty to review, and record cost and failures. Treat retrieved documents as untrusted data. The MCP server security checklist applies the same boundary to tool output; an evaluator must never execute instructions found in the corpus.

Use multiple independent judgments for high-impact samples. Agreement does not prove truth, but disagreement identifies cases requiring human attention. Preserve the human decision and explanation so a future model upgrade can be recalibrated against the same standard.

How does offline evaluation connect to production?

After the offline gate, run the candidate in shadow mode on authorized, redacted traffic without affecting responses. Compare ranking differences, latency, errors, and resource use. Then use a small A/B cohort and monitor successful search, citation opening, reformulation, escalation, latency, and answer-level outcomes.

Clicks are not sufficient relevance labels because rank position and interface design affect them. User satisfaction cannot isolate retrieval from generation. Use online metrics as another evidence layer, not a replacement for qrels and per-query analysis.

Run the frozen benchmark whenever corpus preprocessing, chunking, embeddings, fusion weights, filters, or rerankers change. Save a per-query diff and review the biggest improvements and regressions. Roll out only when accuracy, critical slices, latency, and context budgets all pass. Keep the previous index and configuration available for rollback.

Release checklist

  • Queries, corpus, qrels, chunking, and index configuration are versioned.
  • Development and frozen release sets are separate.
  • The relevance rubric is explicit and disagreements are adjudicated.
  • k matches the context budget used by generation.
  • Recall, MRR, and nDCG definitions match the retrieval job.
  • Reports include per-query and critical-slice results, not only averages.
  • Latency, tokens, external calls, errors, and deduplication are measured.
  • Unanswerable, permission, time-sensitive, multi-hop, and long-tail slices exist.
  • Any LLM judge is calibrated, and corpus instructions are treated as untrusted data.
  • Offline, shadow, online, and rollback evidence form one release path.

Common questions

Can a team start with only a few dozen queries?

Yes. Cover the highest-value intents and known failures first. A small, carefully judged, versioned set is more useful than thousands of easy questions generated from headings.

Is higher Recall@k always better?

It helps when other conditions are equal, but increasing k adds noise, tokens, and latency. Evaluate recall with ranking quality and generation’s actual context budget.

How often should the benchmark change?

Create a new version when the corpus, product intents, or failure distribution changes. Retain old versions for regression. Updating a benchmark should add an explainable time slice, not erase history.