A convincing answer is an easy thing to admire and a difficult thing to debug. When a retrieval-augmented generation system gets something wrong, it is tempting to change the prompt or choose another model.
My RAG evaluation notes suggest an earlier question: did the system retrieve the evidence needed to answer at all?
That splits one vague quality problem into two observable stages.
Two stages, two kinds of failure
Retrieval selects material from a knowledge base. Generation uses that material to answer a question. Either stage can fail independently.
Imagine asking a learning assistant to compare two techniques. It retrieves an excellent explanation of the first technique and nothing about the second. The response might still sound polished, but improving the wording does not supply the missing evidence.
Conversely, both explanations might be present while the answer misrepresents their differences. That is a different failure. I want to inspect the retrieved passages before deciding which part to change.
Start with a small set of real questions
The course material describes a golden dataset: questions paired with reference answers and categories. For my own notes, I would begin with questions I actually want to ask, then record which passages support the answer.
A useful small set would include a direct fact, a comparison, a question requiring several notes, and a question the collection cannot answer. The last case matters: a system that confidently invents an answer should not pass simply because it always produces text.
This is a proposed evaluation approach, not a benchmark I have already run against my Vault.
Ranking and coverage tell different stories
Mean reciprocal rank asks how early the first relevant result appears. If the first useful result is second, the reciprocal rank for that query is one half. Averaging across queries gives MRR; a query with no relevant retrieved result contributes zero.
That helps with ranking, but a comparison may need several pieces of evidence. Finding one useful passage first does not prove that all the necessary passages were found.
I also need a clear definition of relevance. The notebook uses keyword checks as a practical shortcut. A keyword can occur in an irrelevant passage, or be absent from a useful paraphrase. Keyword coverage is a diagnostic signal, not a guarantee of correctness.
The distinction between precision, recall, and ranked retrieval measures is covered in Stanford’s Introduction to Information Retrieval. In particular, finding any relevant item is different from retrieving all the relevant evidence.
Change one thing and inspect the failures
Chunk size is a good example of a parameter with competing effects. A small chunk may isolate a useful fact but lose its explanation. A large chunk can preserve context while mixing several subjects.
The course notes compare different configurations. I should treat those results as course-specific observations, not a universal recipe or my own performance measurements.
For my system, I would save a baseline, change one variable, compare both retrieval and answers, and inspect cases that got worse. An LLM judge could help review answers, but its scores would still need spot-checking against the evidence.
The habit I want to keep is simple: when an answer is wrong, examine what the model was given before asking it to write differently.
Adapted with AI assistance from my Week 5 RAG evaluation notes. Examples here are illustrative, and no Vault retrieval benchmark was executed for this article.