Retrieval works. Now connect it to the model and get answers.
The prompt is the whole job
The default behaviour you want to prevent is the model answering from its training data and presenting it as though it came from your documents. The prompt has to make the boundary explicit:
PROMPT = """Answer the question using only the context below.
Rules:
- If the context does not contain the answer, say "I could not find
that in the provided documents." Do not use outside knowledge.
- Quote the relevant wording where it helps.
- Cite the source filename for each claim, in square brackets.
Context:
{context}
Question: {question}
Answer:"""
The second rule is the important one. Without an explicit instruction and an explicit escape hatch, models fill gaps rather than admitting one exists.
The complete script
import ollama
MODEL = "llama3.1:8b"
def answer(question: str, k: int = 5) -> str:
hits = collection.query(
query_embeddings=[embed_query(question)],
n_results=k,
)
context = "\n\n---\n\n".join(
f"[{meta['source']}]\n{text}"
for text, meta in zip(hits["documents"][0], hits["metadatas"][0])
)
response = ollama.chat(
model=MODEL,
messages=[{
"role": "user",
"content": PROMPT.format(context=context, question=question),
}],
options={"temperature": 0.1},
)
return response["message"]["content"]
if __name__ == "__main__":
while True:
q = input("\nQuestion (or 'quit'): ").strip()
if q.lower() in {"quit", "exit", ""}:
break
print("\n" + answer(q))
That is a complete, working, entirely local RAG system in about thirty lines on top of what you already built.
Why temperature 0.1
Temperature controls randomness. For creative writing you want some. For "tell me what this document says" you want as little as possible — the same question should give the same answer, and invention is the failure mode you are trying to avoid.
Fitting the context
Five chunks of 600 words is roughly 4,000 tokens before the question and
instructions. Comfortable for most models, but watch it if you raise
k — overflowing the window causes silent truncation, and what gets
cut is usually the end of your context rather than anything you would
choose.
Streaming the answer
For anything interactive, stream so the user sees progress:
stream = ollama.chat(
model=MODEL,
messages=[{"role": "user", "content": PROMPT.format(context=context, question=question)}],
options={"temperature": 0.1},
stream=True,
)
for part in stream:
print(part["message"]["content"], end="", flush=True)
A local model on CPU takes a while to produce a long answer. Streaming makes that feel like thinking rather than hanging.