The system works on your machine. This chapter covers what changes when other people need it.
When to leave Chroma
Chroma is fine until roughly one of these is true:
- More than a few hundred thousand chunks.
- Several processes need to write to the index at once.
- You need it on a server rather than a laptop.
- You want proper filtering, sharding or replication.
Qdrant is the usual next step — one container, a similar interface:
docker run -d -p 6333:6333 -v qdrant_storage:/qdrant/storage --name qdrant qdrant/qdrant
pgvector is the better choice if you already run Postgres, because it keeps your vectors and your application data in one database with one backup story. That operational simplicity usually outweighs raw performance.
Keeping the index current
The content hash from chapter four makes incremental updates cheap. Run it on a schedule:
# re-index changed files nightly
0 2 * * * cd /srv/rag && /usr/bin/python3 index.py --incremental
Two things that bite later: deleted documents stay in the index unless you remove them, and will keep being cited as sources for files that no longer exist. And moved files look like a delete plus an add, which is correct but means a reindex.
Serving it to other people
Put a small API in front of it rather than giving people the script. Whatever you use, two rules:
Access control is your job, not the vector store's. A vector database has no concept of who may see which document. If some documents are restricted, filter by permission at query time using the metadata — and test that filter, because a leak here means someone reads a file they should not.
Keep the model local. The easy mistake when scaling is to swap the local model for a hosted API "just for the hard questions". The moment you do, every retrieved passage — the contents of your private documents — is sent to a third party. That is the entire guarantee gone, and it tends to happen quietly in a config change rather than as a decision.
If you are putting this on a server, our VPS tutorial covers the firewall, TLS and hardening that should go with it.
What this has cost you
Nothing per query, permanently. The model and the embedder are on disk, the index is a directory, and the whole thing runs on hardware you already own. A thousand questions a day costs the same as ten.
That economics is why local RAG is worth the setup effort even when a hosted service would be marginally better: you can afford to run it over everything, all the time, without watching a meter.
Where to go next
- Add a reranker for sharper retrieval on hard queries.
- Try a larger model for generation while keeping retrieval unchanged — with the test set from chapter eight you can measure whether it actually helped.
- Index something structured, like a codebase or a ticket archive, where splitting on real boundaries beats fixed-size chunks.