Sunday, 22 November 2026 Press Releases Newsletter RSS
Breaking
WhatsApp Business Account Hacked in Pakistan: Full Recovery Guide Special Technology Zones Authority (STZA) & IT Park Expansion in Pakistan: Islamabad, Karachi, and Lahore Ground Report Pakistan Software Export Board & Industry Consortium Launch 'TechBridge Pakistan 2026' to Connect 500 Startups With Global Markets Global Tech Safety Summit 2026: International Standards for Frontier AI and Data Security Announced WhatsApp Business Account Hacked in Pakistan: Full Recovery Guide Special Technology Zones Authority (STZA) & IT Park Expansion in Pakistan: Islamabad, Karachi, and Lahore Ground Report Pakistan Software Export Board & Industry Consortium Launch 'TechBridge Pakistan 2026' to Connect 500 Startups With Global Markets Global Tech Safety Summit 2026: International Standards for Frontier AI and Data Security Announced
WritePures

This chapter is unglamorous and it decides whether the whole system works. Bad chunking cannot be rescued by a better model.

Getting the text out

Install what you need:

pip install pypdf python-docx markdown-it-py

A minimal extractor:

from pathlib import Path
from pypdf import PdfReader
import docx

def read_file(path: Path) -> str:
    if path.suffix.lower() == ".pdf":
        return "\n".join(page.extract_text() or "" for page in PdfReader(path).pages)
    if path.suffix.lower() == ".docx":
        return "\n".join(p.text for p in docx.Document(path).paragraphs)
    return path.read_text(encoding="utf-8", errors="ignore")

Two warnings worth having now rather than later. Scanned PDFs contain no text — they are images, and you will get empty strings until you run OCR. And tables extract badly, often collapsing into unreadable runs of numbers. Check a sample of your extracted text before indexing thousands of files.

How big should a chunk be?

The trade-off is simple. Too small and a chunk lacks the context to be understood on its own. Too large and the embedding becomes an average of several topics, which matches nothing precisely.

A reasonable default: 500–800 tokens, roughly 400–600 words, with 10–15% overlap between neighbouring chunks.

The overlap matters more than it looks. Without it, a sentence that straddles a boundary gets cut in half and neither chunk makes sense.

def chunk(text: str, size: int = 2000, overlap: int = 250):
    """Character-based chunking. Crude, and good enough to start."""
    chunks, start = [], 0
    while start < len(text):
        end = start + size
        # Prefer to break at a paragraph, then a sentence, then anywhere.
        if end < len(text):
            for sep in ("\n\n", ". ", " "):
                cut = text.rfind(sep, start, end)
                if cut > start:
                    end = cut + len(sep)
                    break
        chunks.append(text[start:end].strip())
        start = end - overlap
    return [c for c in chunks if c]

Split on structure when you have it

Fixed-size chunking is the fallback. If your documents have real structure, use it: split a contract on clauses, a manual on headings, a codebase on functions. A chunk that corresponds to one actual section beats an arbitrary 2,000-character window every time.

Keep metadata with every chunk

This is the step people skip and regret, because without it you cannot cite sources in chapter seven.

records = []
for path in Path("documents").rglob("*"):
    if path.is_file():
        for i, piece in enumerate(chunk(read_file(path))):
            records.append({
                "id": f"{path.name}-{i}",
                "text": piece,
                "source": str(path),
                "chunk_index": i,
            })

Store the filename at minimum. Page number, section heading and date are worth adding if you have them — they let you answer "where did this come from" and filter searches later.

Check before you commit

Print a few chunks and read them. Does each one make sense on its own? If a chunk is half a sentence, or three unrelated topics, fix the chunking now. You are about to embed every one of these, and re-running after a fix costs the whole import time again.