This chapter is unglamorous and it decides whether the whole system works. Bad chunking cannot be rescued by a better model.
Getting the text out
Install what you need:
pip install pypdf python-docx markdown-it-py
A minimal extractor:
from pathlib import Path
from pypdf import PdfReader
import docx
def read_file(path: Path) -> str:
if path.suffix.lower() == ".pdf":
return "\n".join(page.extract_text() or "" for page in PdfReader(path).pages)
if path.suffix.lower() == ".docx":
return "\n".join(p.text for p in docx.Document(path).paragraphs)
return path.read_text(encoding="utf-8", errors="ignore")
Two warnings worth having now rather than later. Scanned PDFs contain no text — they are images, and you will get empty strings until you run OCR. And tables extract badly, often collapsing into unreadable runs of numbers. Check a sample of your extracted text before indexing thousands of files.
How big should a chunk be?
The trade-off is simple. Too small and a chunk lacks the context to be understood on its own. Too large and the embedding becomes an average of several topics, which matches nothing precisely.
A reasonable default: 500–800 tokens, roughly 400–600 words, with 10–15% overlap between neighbouring chunks.
The overlap matters more than it looks. Without it, a sentence that straddles a boundary gets cut in half and neither chunk makes sense.
def chunk(text: str, size: int = 2000, overlap: int = 250):
"""Character-based chunking. Crude, and good enough to start."""
chunks, start = [], 0
while start < len(text):
end = start + size
# Prefer to break at a paragraph, then a sentence, then anywhere.
if end < len(text):
for sep in ("\n\n", ". ", " "):
cut = text.rfind(sep, start, end)
if cut > start:
end = cut + len(sep)
break
chunks.append(text[start:end].strip())
start = end - overlap
return [c for c in chunks if c]
Split on structure when you have it
Fixed-size chunking is the fallback. If your documents have real structure, use it: split a contract on clauses, a manual on headings, a codebase on functions. A chunk that corresponds to one actual section beats an arbitrary 2,000-character window every time.
Keep metadata with every chunk
This is the step people skip and regret, because without it you cannot cite sources in chapter seven.
records = []
for path in Path("documents").rglob("*"):
if path.is_file():
for i, piece in enumerate(chunk(read_file(path))):
records.append({
"id": f"{path.name}-{i}",
"text": piece,
"source": str(path),
"chunk_index": i,
})
Store the filename at minimum. Page number, section heading and date are worth adding if you have them — they let you answer "where did this come from" and filter searches later.
Check before you commit
Print a few chunks and read them. Does each one make sense on its own? If a chunk is half a sentence, or three unrelated topics, fix the chunking now. You are about to embed every one of these, and re-running after a fix costs the whole import time again.