Memory bandwidth predicts local AI speed: 23 chips, one model

Every time a local AI model writes a word, it reads its weights from memory. For a 7-billion-parameter model compressed to 4 bits, that is about 3.75 GB of data per token. So a simple rule should hold: the faster the memory, the faster the answer. We tested that rule on four chips in our own two machines and checked it against published results for 19 more.

The rule, and why it works

Divide a machine’s memory bandwidth by the amount of data the model reads per token and you get the fastest it could possibly write. Our RTX 4060 has 272 GB/s of memory bandwidth, so with Llama 2 7B (3.75 GB read per token) the limit is 272 / 3.75 = about 72 tokens per second. We measured 65.5, or 90% of the limit.

The same arithmetic works for a processor with no graphics card. Our Ryzen 9 7900X has dual-channel DDR5-6400 memory, a theoretical 102.4 GB/s, which allows about 27 tokens per second. We measured 15.5, or 57% of the limit. Desktop processors reach a smaller share because they cannot keep their memory as busy as a graphics card can.

Llama 2 7B on our desktop, with and without the graphics card

Model	Where it ran	Reading the prompttokens/s	Writing the answertokens/s	Test
Llama 2 7B Q4_0	NVIDIA GeForce RTX 4060 8 GB Whole model on GPU	2,716	65.5	T0002
Llama 2 7B Q4_0	AMD Ryzen 9 7900X (CPU only) CPU only	134	15.5	T0001

23 chips, one model

We chose Llama 2 7B at 4 bits (Q4_0) for our reference test because it is the model the llama.cpp community has used to benchmark hardware for years. That means we can place our results next to published ones measured the same way, from an M1 MacBook to an RTX 5090.

• Our tests
• Apple Silicon (published)
• PC graphics and AI chips (published)

Hover a point for its value. Every point is listed in the table below.

The points hug a straight line: across an 80-fold range of bandwidth, writing speed rises almost in step with it. Most reach between 55% and 85% of their theoretical limit.

What the exceptions tell you

• The very fastest chips fall further behind. An RTX 5090 (1,792 GB/s) writes at 264 tokens per second, about 55% of its limit, and Apple’s Ultra chips reach under 50%. With this small model, they finish reading the weights so quickly that computation and software overheads start to dominate. With larger models they get closer to their limit.
• Software matters. The Intel Arc A770 has twice the bandwidth of our RTX 4060 but wrote slower in its published result (45 versus 65.5 tokens per second). Drivers and how well a chip’s backend is optimized can cost more than half the potential speed.
• Unified-memory machines land in the middle. AMD’s Ryzen AI Max+ 395 (256 GB/s) wrote at 48 tokens per second and NVIDIA’s DGX Spark (273 GB/s) at 58.5, 70% and 80% of their limits. What they offer is not speed but capacity: up to 128 GB, enough for models no consumer graphics card can hold.

The rule of thumb

For buying decisions, this is accurate enough:

Tokens per second ≈ 0.7 × memory bandwidth (GB/s) ÷ model size (GB)

Use about 0.85 for a modern NVIDIA graphics card and about 0.55 for a desktop processor on its own. Slow memory is easier to keep busy: our budget mini PC reached about 70% on its processor and 85% on its integrated graphics. Two refinements make it work for current models:

1. For mixture-of-experts models, use the active size, not the file size. gpt-oss 20B is a 12 GB file, but only about 2.6 GB is read per token, which is why our Ryzen 9 7900X alone writes it at 21.8 tokens per second, faster than the much smaller Qwen3.5 9B (10.4).
2. Long conversations add reading. The conversation memory (KV cache) is read for every token too. On our RTX 4060, Llama 2 7B slowed from 65.5 to 43.3 tokens per second with 4,096 tokens of conversation, because that older model stores a lot per token. Newer models store far less and barely slow down.

Our Can it run? calculator applies all of this for you, using each model’s real file size, active size and conversation memory, read directly from the model files.

What this means when you buy

• Memory capacity decides which models you can run; bandwidth decides how fast. Check both.
• For models up to about 9 billion parameters, an 8 GB graphics card like the RTX 4060 is already fast (45 to 68 tokens per second in our tests).
• For bigger models, you are choosing between a graphics card with more video memory (fast, but typically 8 to 32 GB on consumer cards) and a unified-memory machine with 64 to 128 GB (slower, but it fits much larger models).
• Spec sheets that lead with NPU “TOPS” are measuring something else. For local language models, look for the memory bandwidth figure.

The slow end: a $369 mini PC

Our ACEMAGIC K1 has a 4-core Ryzen 3 4300U and ships with a single 16 GB stick of DDR4-2666. With one stick, memory runs on one channel, so it has only 21.3 GB/s, the lowest bandwidth in this study. It lands where the rule predicts: 3.9 tokens per second on its processor (69% of its limit) and 4.8 on its integrated graphics (85%).

Llama 2 7B on the ACEMAGIC K1, processor and integrated graphics

Model	Where it ran	Reading the prompttokens/s	Writing the answertokens/s	Test
Llama 2 7B Q4_0	AMD Ryzen 3 4300U (CPU only) CPU only	20.5	3.94	T0003
Llama 2 7B Q4_0	AMD Radeon Graphics, integrated (Ryzen 3 4300U) Whole model on GPU	38.9	4.82	T0004

The lesson for budget machines: check that memory runs in dual channel. The K1 has a second, empty slot, and a matching stick would double its bandwidth to 42.7 GB/s.