You have seen Q4_K_M several times now. This chapter explains
it, because understanding it is what lets you fit a better model onto the
machine you already own.
The idea
A model's weights are numbers. Originally they are stored at high precision — 16 bits each. Quantisation stores them with fewer bits, which makes the file smaller and the model faster, at some cost to quality.
The ratio is unusually favourable. Going from 16-bit to 4-bit cuts memory by roughly three quarters, while quality loss on most everyday tasks is small enough that many people cannot identify which they are using in a blind test.
That is why essentially everyone runs quantised models locally, and why the memory table in chapter two assumed it.
Reading the names
A name like Q4_K_M breaks down as:
- Q4 — 4 bits per weight.
- K — the newer "k-quant" method, which allocates precision more intelligently across the model.
- M — medium, within that method. You will also see S (small) and L (large).
What to choose
| Level | Size vs 16-bit | Use when |
|---|---|---|
| Q8 | ~50% | You have memory to spare and want maximum fidelity |
| Q6_K | ~38% | A good compromise if Q4 feels lossy on your task |
| Q4_K_M | ~27% | The default. Start here. |
| Q3_K_M | ~21% | Only to fit a larger model that otherwise will not load |
| Q2_K | ~16% | Rarely worth it; quality degrades noticeably |
The decision that actually matters
Given a fixed amount of memory, you usually face a choice between:
- A smaller model at high precision — say 7B at Q8.
- A larger model at lower precision — say 13B at Q4_K_M.
The general experience is that the larger model at Q4 wins. Parameter count buys more capability than precision does, down to about Q4. Below Q4 that reverses and the quality loss becomes obvious.
So the practical rule: pick the largest model that fits at Q4_K_M, rather than a smaller one at higher precision.
Choosing a specific quantisation in Ollama
Ollama's default tags are already quantised sensibly, which is why you have not had to think about this. When you want a specific level, the library lists tags for it:
# explicit quantisation, where the library offers it
ollama run llama3.1:8b-instruct-q8_0
# confirm what you actually got
ollama show llama3.1:8b
ollama show prints the quantisation of the downloaded model.
Run it when a model performs unexpectedly well or badly — you may not have the
file you assumed.
Where GGUF comes in
You will see GGUF mentioned constantly. It is the file format these quantised models are distributed in, originating with the llama.cpp project that most local runtimes are built on.
For this tutorial you do not need to handle GGUF files directly — Ollama does it. It matters when you want a model from Hugging Face that is not in Ollama's library, which is a reasonable next step once you are comfortable.