Build Retrieval-Augmented Generation That You Can Evaluate

Measure whether the right evidence was found before judging whether the answer sounds right.

A learner asks how long fragile equipment can be borrowed. The assistant retrieves the first paragraph of the loan policy and answers, “Seven days.” The next paragraph says fragile equipment has a two-day limit.

The answer cited a real source. It missed the controlling exception.

Retrieval-augmented generation, commonly called RAG, combines information retrieval with generated answers. Its value depends on the quality of both stages and the connection between them. A fluent answer cannot compensate for missing evidence.

Define the retrieval problem

Name the collection, users, question types, and acceptable outcomes. Our fictional system answers questions about current public hours and loans. It must keep staff records outside public retrieval and avoid using retired policies for current questions.

The original RAG research paper studied combining a pretrained generator with retrieved information for knowledge-intensive tasks. Modern applications use many variations, including different retrievers, indexes, and ways of constructing context.

For a first implementation, make the evidence path inspectable. A small keyword baseline can reveal whether document preparation and access filtering work before adding embeddings or a more complex ranking model.

Prepare sources before splitting them

Retain document identity, version, status, access scope, and location information. Check extracted text against the original, especially tables, footnotes, and exceptions.

Chunking divides a document into smaller units for retrieval. A chunk should be large enough to preserve a useful meaning and small enough to avoid burying that meaning in unrelated text. Paragraph boundaries are a reasonable teaching baseline, but they can separate a general rule from its exception.

Overlap can preserve nearby context, yet it also creates duplicate passages and increases storage and prompt size. Another approach retrieves a small passage and then includes its parent section or adjacent paragraphs. Evaluate the choice on actual questions rather than assuming one chunk size is universally best.

The example below assigns paragraph identifiers so the evidence can be traced back to its document.

Run a small retrieval baseline

Save this as retrieval_lab.py and run it with Python 3.9 or later. It uses a simple count of shared words, current-status filtering, and trusted role filtering. It includes all synthetic documents and checks.

This is a tested retrieval core. It does not call a generative model, compute embeddings, or claim production-grade ranking quality.

import re

DOCUMENTS = [
    {"id": "HOURS", "version": 2, "status": "current", "access": "public",
     "text": "Saturday opening hours are 10 a.m. to noon."},
    {"id": "LOANS", "version": 1, "status": "current", "access": "public",
     "text": "Standard loan duration is seven calendar days.\n\nFragile equipment loan duration is two calendar days. This exception overrides the standard duration."},
    {"id": "OLD-HOURS", "version": 1, "status": "retired", "access": "public",
     "text": "Saturday opening hours are 9 a.m. to noon."},
    {"id": "STAFF", "version": 1, "status": "current", "access": "staff",
     "text": "The internal staff contact is Person One."},
]
STOP = {"the", "is", "are", "a", "an", "for", "of", "to", "on", "how", "what", "can", "i", "and", "in"}


def words(text):
    return set(re.findall(r"[a-z0-9]+", text.casefold())) - STOP


def retrieve(question, trusted_role, documents=DOCUMENTS, limit=2):
    if trusted_role not in {"public", "staff"}:
        return []
    candidates = []
    for document in documents:
        if document.get("status") != "current":
            continue
        access = document.get("access")
        if access not in ("public", "staff"):
            continue
        if access == "staff" and trusted_role != "staff":
            continue
        for number, paragraph in enumerate(document["text"].split("\n\n"), 1):
            score = len(words(question) & words(paragraph))
            if score:
                candidates.append({"id": f'{document["id"]}:{number}',
                                   "version": document["version"], "text": paragraph,
                                   "score": score})
    return sorted(candidates, key=lambda row: (-row["score"], row["id"]))[:limit]


def main():
    fixtures = [
        ("Saturday opening hours", {"HOURS:1"}),
        ("Fragile equipment loan duration", {"LOANS:1", "LOANS:2"}),
    ]
    for question, gold in fixtures:
        found = retrieve(question, "public")
        ids = {row["id"] for row in found}
        recall = len(ids & gold) / len(gold)
        assert recall == 1.0
        print(question, sorted(ids), "evidence recall", recall)
    assert retrieve("refund policy", "public") == []
    assert retrieve("internal staff contact", "public") == []
    assert retrieve("internal staff contact", "staff")[0]["id"] == "STAFF:1"
    assert retrieve("Saturday opening hours", None) == []
    conflicting = DOCUMENTS + [{"id": "OTHER-HOURS", "version": 1,
        "status": "current", "access": "public", "text": "Saturday opening hours are 11 a.m. to noon."}]
    assert {row["id"] for row in retrieve("Saturday opening hours", "public", conflicting)} == {"HOURS:1", "OTHER-HOURS:1"}
    for access in ("finance", None, []):
        unknown = {"id": "UNKNOWN", "version": 1, "status": "current",
                   "access": access, "text": "Restricted audit details."}
        assert retrieve("audit details", "staff", [unknown]) == []
    del unknown["access"]
    assert retrieve("audit details", "staff", [unknown]) == []
    print("PASS: evidence retrieval, missing topic, closed access filtering, and conflict visibility")


if __name__ == "__main__":
    main()

The local checks passed during preparation. The Saturday question retrieved the current hours passage. The fragile-equipment question retrieved both the general rule and its exception. A refund question returned no evidence. Public retrieval excluded the staff record, while the staff fixture could retrieve it. A conflicting current hours passage remained visible alongside the original. Additional checks deny unknown, missing, null, and malformed access labels even for staff. The filter permits only the two explicitly defined access levels; a staff role is not universal permission for every label that might appear later.

The trusted role is a test fixture. A real application must derive it from authenticated access records. A model-supplied role string must not grant access.

Compare retrieval methods against the same questions

Keyword retrieval can work well for exact terms, names, and identifiers. Embedding retrieval represents text as vectors and compares their positions, which can help with paraphrases. A hybrid system combines different retrieval signals. A reranker examines candidate passages more closely before choosing what enters the answer context.

Each method can fail differently. Keyword search may miss “When must I bring this back?” when the policy says “loan duration.” Vector search may return conceptually similar but legally or operationally different rules. A reranker cannot recover a passage excluded from its candidate set.

Keep an evaluation set separate from the examples used to design the retriever. Compare the same questions, corpus version, access scope, and evidence labels. The fixtures in this article are development checks, not an independent held-out benchmark.

When adding a reranker, first retrieve a candidate pool, then rerank it, then select the final passages. Record both the candidate and final sets so you can tell whether a relevant exception was never found or was discarded later.

Measure evidence coverage

For a question with a known set of required passages, evidence recall is the number of required passages retrieved divided by the number required. The fragile-loan fixture labels two passages as useful together: the standard rule and the exception. Retrieving both gives recall 2/2, or 1.0, for that fixture.

Precision asks how much retrieved material is relevant. Retrieving the whole archive may increase recall while overwhelming the answer stage with unrelated or conflicting text. Ranking measures can also assess whether useful evidence appears early enough to enter a limited context window.

The right labels depend on the task. Sometimes one passage is sufficient. Sometimes an answer needs several documents, a definition, and an exception. Record that distinction in the answer key rather than treating every question as a one-passage lookup.

Build the answer from the actual evidence

For the fragile-equipment question, an acceptable answer is: “Fragile equipment has a two-calendar-day loan limit. This overrides the standard seven-day duration [LOANS v1, paragraph 2; context in paragraph 1].”

The answer identifies the exception and its scope. It does not infer that the coordinator has approved an extension or that every item is eligible.

Pass the retrieved passages, their identifiers, and a clear answer contract to the generator. Require citations for factual claims and a useful statement of missing evidence when the collection is insufficient. Then check the final claims against the cited text.

A citation string alone is not enough. The cited passage must have been retrieved, belong to the authorized source set, and support the claim. A model can invent a citation just as it can invent a fact.

Preserve conflicts and missing answers

The conflict fixture contains two current public passages with different Saturday opening times. Retrieval should expose both. Resolving authority requires information about ownership, applicability, or supersession that the fixture does not supply.

The generator should report the conflict and seek the appropriate source owner. Choosing the first passage or averaging the times would invent a resolution.

For the refund question, no retrieved evidence means the collection does not establish the policy. The assistant can say what is missing and suggest where an authorized user might verify it. It should not write a plausible refund policy from general business knowledge when the task requires this center's actual rule.

Evaluate retrieval and generation separately

Use two views of every failure. First, did the retrieval stage return the required authorized evidence? Second, did the answer accurately use that evidence without unsupported additions?

A correct answer from memory can hide a retrieval failure. An incorrect answer with the right passages indicates a different problem. Keep both measurements so improvements target the responsible stage.

Useful failure categories include missing source text, wrong version, access-filter error, poor candidate retrieval, reranking loss, misunderstood exception, unsupported synthesis, and invalid citation.

Maintain the index as a derivative of the sources

When a source changes, determine how its new version enters the index and how the old version leaves current answers. When access changes, check caches and stored summaries as well as the primary index. When a document is deleted, define how deletion propagates.

Retrieved content remains untrusted as an instruction source. A passage saying “ignore restrictions” should not broaden tool access. OWASP's RAG security guidance discusses risks across ingestion, retrieval, and generated output.

Maintain a compact evaluation set that includes a known updated answer and a denied record. Re-run those cases after changes to parsing, chunking, filters, indexes, or models.

A reusable prompt

Design a retrieval-and-answer workflow for this collection and user scope. Preserve source identity, version, and passage locations. Compare a simple baseline with proposed retrieval and reranking changes on the same held-out questions. Evaluate evidence coverage separately from answer correctness. Require support for citations and explicit handling of missing or conflicting evidence.

For students: make the exception part of the exercise

Students can use approved course readings to label the passages needed for a question. Include definitions and exceptions, not just the first paragraph containing a keyword.

Ask the assistant to answer from those passages, then check every cited claim. Practice a related question without assistance to verify learning. Do not place restricted assessments or classmates' private work into a shared index.

Practice: retrieve the rule and its exception

Run the example. Reduce the result limit to one for the fragile-loan question and inspect which evidence remains. Explain whether the surviving passage alone supports the answer and what context has been lost. Then add a paraphrased question and record how the keyword baseline behaves.

Completion check: A correct-looking answer is accepted only when authorized supporting evidence was retrieved and the answer uses it accurately.

Stretch: Compare paragraph chunks with parent-section retrieval on a held-out question set. Report both evidence coverage and the amount of irrelevant text added.