This is the chapter that saves you an afternoon. Almost every frustrated "local models are useless" post comes from someone who tried to run a model their machine could not hold.
The only rule you need
At the common Q4_K_M quantisation, budget roughly:
0.5 to 0.6 GB of memory per billion parameters, then add a few GB for context and the operating system.
So:
| Model size | Weights at Q4_K_M | Comfortable total memory |
|---|---|---|
| 3B | ~2 GB | 8 GB |
| 7B / 8B | ~4.4–6 GB | 16 GB |
| 13B / 14B | ~8–9 GB | 16–24 GB |
| 32B | ~19–20 GB | 32 GB |
| 70B | ~40–48 GB | 64 GB or two GPUs |
Sources vary a little on overhead, so treat these as close approximations rather than exact thresholds. The shape is what matters.
What this means in practice
8 GB RAM: 3B models comfortably, a 7B at a push if little else is running. Start here and expect to want more.
16 GB RAM: the sweet spot. 7B and 8B models run well, 13B is reachable. This covers most laptops sold in the last few years and most of what this tutorial does.
24 GB VRAM (RTX 4090 / 3090): fast 13B, workable 32B. Note that a 70B at Q4_K_M does not fit on a 24 GB card — that is a common and expensive misunderstanding.
Apple Silicon: unusually good at this, because the CPU and GPU share one memory pool. An M-series Mac with 32 GB can run models that would need a dedicated 32 GB GPU on a PC. A 128 GB machine can handle 70B at Q4.
CPU or GPU?
You do not need a GPU. You will want one.
On CPU, a 7B model produces a few words per second — usable for drafting and batch work, irritating for conversation. On a reasonable GPU the same model is faster than you can read.
For large models on CPU the picture is worse: a 70B at Q4_K_M on system RAM alone is commonly reported under 3 tokens per second and often between 0.5 and 1.5. That is technically working and practically unusable for anything interactive.
One widely repeated guideline when you do have a GPU: keep at least twice your VRAM in system RAM, so loading and offloading do not thrash.
Disk
Models are large files and you will collect more than you intend. Budget 10 GB for your first, and 50–100 GB if you plan to compare several. An SSD matters for load time, not for generation speed.
Check what you have
On macOS or Linux:
# total RAM, macOS
sysctl hw.memsize
# total RAM, Linux
free -h
# free disk space
df -h
On Windows, in PowerShell:
Get-CimInstance Win32_ComputerSystem | Select-Object TotalPhysicalMemory
Get-PSDrive C
Note your RAM figure. In the next chapter you will install the runtime, and in chapter four you will choose a model size based on exactly this number.