Saturday, 21 November 2026 Press Releases Newsletter RSS
Breaking
WhatsApp Business Account Hacked in Pakistan: Full Recovery Guide Special Technology Zones Authority (STZA) & IT Park Expansion in Pakistan: Islamabad, Karachi, and Lahore Ground Report Pakistan Software Export Board & Industry Consortium Launch 'TechBridge Pakistan 2026' to Connect 500 Startups With Global Markets Global Tech Safety Summit 2026: International Standards for Frontier AI and Data Security Announced WhatsApp Business Account Hacked in Pakistan: Full Recovery Guide Special Technology Zones Authority (STZA) & IT Park Expansion in Pakistan: Islamabad, Karachi, and Lahore Ground Report Pakistan Software Export Board & Industry Consortium Launch 'TechBridge Pakistan 2026' to Connect 500 Startups With Global Markets Global Tech Safety Summit 2026: International Standards for Frontier AI and Data Security Announced
WritePures

Retrieval works. Now connect it to the model and get answers.

The prompt is the whole job

The default behaviour you want to prevent is the model answering from its training data and presenting it as though it came from your documents. The prompt has to make the boundary explicit:

PROMPT = """Answer the question using only the context below.

Rules:
- If the context does not contain the answer, say "I could not find
  that in the provided documents." Do not use outside knowledge.
- Quote the relevant wording where it helps.
- Cite the source filename for each claim, in square brackets.

Context:
{context}

Question: {question}

Answer:"""

The second rule is the important one. Without an explicit instruction and an explicit escape hatch, models fill gaps rather than admitting one exists.

The complete script

import ollama

MODEL = "llama3.1:8b"

def answer(question: str, k: int = 5) -> str:
    hits = collection.query(
        query_embeddings=[embed_query(question)],
        n_results=k,
    )

    context = "\n\n---\n\n".join(
        f"[{meta['source']}]\n{text}"
        for text, meta in zip(hits["documents"][0], hits["metadatas"][0])
    )

    response = ollama.chat(
        model=MODEL,
        messages=[{
            "role": "user",
            "content": PROMPT.format(context=context, question=question),
        }],
        options={"temperature": 0.1},
    )

    return response["message"]["content"]

if __name__ == "__main__":
    while True:
        q = input("\nQuestion (or 'quit'): ").strip()
        if q.lower() in {"quit", "exit", ""}:
            break
        print("\n" + answer(q))

That is a complete, working, entirely local RAG system in about thirty lines on top of what you already built.

Why temperature 0.1

Temperature controls randomness. For creative writing you want some. For "tell me what this document says" you want as little as possible — the same question should give the same answer, and invention is the failure mode you are trying to avoid.

Fitting the context

Five chunks of 600 words is roughly 4,000 tokens before the question and instructions. Comfortable for most models, but watch it if you raise k — overflowing the window causes silent truncation, and what gets cut is usually the end of your context rather than anything you would choose.

Streaming the answer

For anything interactive, stream so the user sees progress:

stream = ollama.chat(
    model=MODEL,
    messages=[{"role": "user", "content": PROMPT.format(context=context, question=question)}],
    options={"temperature": 0.1},
    stream=True,
)

for part in stream:
    print(part["message"]["content"], end="", flush=True)

A local model on CPU takes a while to produce a long answer. Streaming makes that feel like thinking rather than hanging.