Embeddings are generated by a different model from the one that writes answers. It is smaller, faster, and its only job is turning text into vectors.
Pulling an embedding model
ollama pull nomic-embed-text
nomic-embed-text is the sensible default for English: 768 dimensions, around 274 MB, and by a wide margin the most used local embedding model.
The main alternative, mxbai-embed-large, produces 1,024 dimensions and is strong on short English passages — but note its 512-token limit, which is smaller than the chunks suggested in the last chapter. If you use it, chunk shorter.
Newer options exist, including a v2 nomic model with different dimensions and limits. Check the library and the model card rather than assuming the numbers above apply to whatever tag you pull.
Generating a vector
pip install ollama
import ollama
def embed(text: str) -> list[float]:
result = ollama.embeddings(model="nomic-embed-text", prompt=text)
return result["embedding"]
vector = embed("The agreement may be terminated on thirty days notice.")
print(len(vector)) # 768
The prefix requirement — read this one
This is the single most common silent failure in local RAG, and it produces a system that works just well enough that you do not realise it is broken.
The nomic models are trained to expect a prefix telling them whether they are embedding a document or a search query:
def embed_document(text: str) -> list[float]:
return embed(f"search_document: {text}")
def embed_query(text: str) -> list[float]:
return embed(f"search_query: {text}")
Use search_document: when indexing and
search_query: when searching. Omit them and retrieval still
returns results — just noticeably worse ones, with no error to tell you
why.
mxbai has its own query instruction convention. Whatever model you pick, read its card for this specifically.
Embedding the whole collection
from tqdm import tqdm
for record in tqdm(records, desc="Embedding"):
record["vector"] = embed_document(record["text"])
Expect this to take a while — on CPU, budget a few thousand chunks per hour. It is a one-off cost: only new or changed documents need re-embedding afterwards, which is why storing a content hash per chunk is worth the five lines it takes.
A sanity check before moving on
Embed two sentences that mean the same thing in different words, and one that means something else. Compare them:
import numpy as np
def similarity(a, b):
a, b = np.array(a), np.array(b)
return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
a = embed_document("The contract ends after thirty days notice.")
b = embed_document("Either party may terminate with one month's warning.")
c = embed_document("The office kitchen is cleaned on Fridays.")
print(similarity(a, b)) # should be clearly higher
print(similarity(a, c)) # should be clearly lower
If those two numbers are close together, something is wrong — usually the model tag or a missing prefix. Fix it before indexing ten thousand documents.