Most RAG systems are evaluated by their author asking three questions they already know the answer to, being satisfied, and shipping. Then real users ask different questions and it falls apart.
Build a test set — thirty minutes, once
Write 20–30 real questions people would actually ask, and for each, note which document contains the answer. A plain file is enough:
tests = [
{"q": "What is the notice period for termination?", "expect": "contracts/supplier-agreement.pdf"},
{"q": "Who approves expenses over 50,000?", "expect": "policies/finance-policy.docx"},
{"q": "What is the warranty on the X200?", "expect": "manuals/x200.pdf"},
]
Include questions you expect to fail — things genuinely not in your documents. A system that answers those confidently is worse than one that finds nothing.
Measure the two stages separately
This is the part people skip and it is the only way to know what to fix.
Retrieval recall — did the right document make the top k?
def retrieval_recall(tests, k=5):
hits = 0
for t in tests:
result = collection.query(query_embeddings=[embed_query(t["q"])], n_results=k)
files = {m["source"] for m in result["metadatas"][0]}
if any(t["expect"] in f for f in files):
hits += 1
else:
print(f"MISS: {t['q']}")
return hits / len(tests)
Answer quality — given the right passages, was the answer right? Judge this by reading. Twenty answers takes fifteen minutes and tells you more than any automated metric at this scale.
What the numbers mean
- Low recall, good answers. Fix retrieval: chunking,
hybrid search, a bigger
k. The model is fine. - Good recall, poor answers. Fix the prompt or use a larger model. Retrieval is fine.
- Both poor. Start with retrieval. Fixing generation while the model is being fed the wrong passages is guesswork.
Aim for recall above 0.9 before worrying about anything else. The model cannot answer from a passage it never saw.
Tune in this order
- Chunking. Biggest single lever, and the one people change last.
- Hybrid search. Usually a step change if your documents contain codes, names or numbers.
- k. Cheap to try. Go up until answers stop improving.
- Prompt. Mostly about refusal behaviour and citation format.
- Model size. Last, because it is the most expensive and usually the least effective fix.
Re-run after every change
Keep the test set in version control next to the code and re-run it after each change. RAG tuning is full of changes that improve one question and break three others, and without a test set you will not notice — you will just have a vague sense that it used to be better.