A working pipeline that retrieves the wrong passages produces confident wrong answers. This chapter is about the retrieval quality that determines everything downstream.
How many chunks to return
Three to five is the usual sweet spot.
Too few and you miss the passage containing the answer. Too many and you fill the context window with noise, which both slows generation and measurably degrades answers — models are worse at finding a fact buried in the middle of a long context than at the edges.
Start at 5 and tune on real questions.
Where pure vector search fails
Embeddings capture meaning, which is exactly why they are bad at exact strings.
Search for invoice number INV-2026-4471 and vector search
returns passages that are semantically about invoices — not the one with that
number. Same for product codes, surnames, error codes and legal citations.
The fix is hybrid search: run both a keyword search and a vector search, then merge.
pip install rank-bm25
from rank_bm25 import BM25Okapi
corpus = [r["text"] for r in records]
bm25 = BM25Okapi([doc.lower().split() for doc in corpus])
def hybrid_search(question: str, k: int = 5):
# Keyword side
keyword_scores = bm25.get_scores(question.lower().split())
keyword_top = sorted(range(len(corpus)), key=lambda i: -keyword_scores[i])[:k]
# Vector side
vector_hits = collection.query(query_embeddings=[embed_query(question)], n_results=k)
vector_ids = vector_hits["ids"][0]
# Reciprocal rank fusion: rank position matters, raw scores do not compare
scores = {}
for rank, idx in enumerate(keyword_top):
scores[records[idx]["id"]] = scores.get(records[idx]["id"], 0) + 1 / (60 + rank)
for rank, doc_id in enumerate(vector_ids):
scores[doc_id] = scores.get(doc_id, 0) + 1 / (60 + rank)
return sorted(scores, key=scores.get, reverse=True)[:k]
Reciprocal rank fusion is the standard way to merge two rankings whose scores are not on the same scale. The constant 60 is conventional and not magic.
Filtering with metadata
The metadata you stored in chapter two earns its keep here:
collection.query(
query_embeddings=[embed_query("notice period")],
n_results=5,
where={"source": {"$contains": "contracts/"}},
)
Narrowing to a folder, a date range or a document type before searching is often a bigger quality win than any model change.
Reranking, if you need more
If retrieval is still imprecise, retrieve more candidates than you need — say 20 — then score each one against the question with a small model and keep the best 5. It costs a second and noticeably sharpens results on hard queries.
Test retrieval on its own
This is the discipline that makes debugging possible. Before involving the language model at all, write ten real questions and check whether the right passage appears in the top five.
If it does not, no prompt engineering will save you — the model never sees the answer. Fixing retrieval first, in isolation, is the difference between a system you can improve and one you can only guess at.