Dedicated AI Hardware

The Memory Bandwidth Bottleneck

Article one — performance

Every conversation about local model speed arrives at the same wall. It is not the processor. It is the road the data travels on.

People shopping for a machine to run models ask the wrong question first. They ask how fast the chip is. The honest answer is that the chip is almost never the limiting factor once the model is loaded. The limiting factor is how quickly the machine can feed that chip, and how much of the model has to be dragged across a narrow path before a single word comes out.

This page explains the memory bandwidth bottleneck in plain terms, then converts it into a short list of numbers you can actually check on a specification sheet.

The token loop, in one paragraph

Text generation happens one token at a time. For each token, the model takes the current sequence, computes attention against everything that came before, and then pushes the result through every weight layer in the network to produce a probability distribution over the next token. Then it samples one token, appends it to the sequence, and starts again.

The critical consequence: producing token number four hundred requires reading the entire set of active weights again. Not a cached fragment. The whole working set. Every single token, every single time.

So the throughput ceiling in tokens per second is approximately the memory bandwidth available to the compute units divided by the number of bytes that must be read per token. That is the entire model. If a machine can move five hundred gigabytes per second and a quantized model occupies thirty gigabytes of active weights, the arithmetic ceiling is under seventeen tokens per second before any overhead. Real systems land below that ceiling, never above it.

Arithmetic intensity, or why FLOPs mislead

Peak floating-point throughput is measured in operations per second, and it looks spectacular on a box. It describes the best case, where the machine performs many operations on each byte it loads. Inference during token generation is the opposite case. The model performs a small amount of arithmetic per byte read, because at batch size one there is only one sequence moving through at a time. Nothing is reused.

Hardware designers know this. That is why accelerators for inference are built with enormous memory bandwidth relative to their arithmetic units, and why the ratio, not either number alone, is the interesting figure. A chip with modest compute and extreme bandwidth will beat a chip with enormous compute and modest bandwidth on this specific workload, and it will do so without appearing to work hard.

This is also why adding a second accelerator rarely doubles speed. The two devices split the work, but if they are drawing from the same memory pool or crossing the same narrow interconnect, they wait on the same road.

The interconnect problem

Most desktop accelerators sit on a PCIe slot. That link is fast in burst terms and thin in sustained terms compared with the on-package memory of the accelerator itself. For training, where you transfer a batch and then compute for a while, the link is rarely the wall. For generation, where you are reading weights continuously, the link becomes a bottleneck the moment the model does not fit entirely in device memory.

When a model overflows device memory, the system must stream weights from host memory to the device for every layer, for every token. That is when tokens per second collapses from a respectable rate to something measured in single digits and seconds per word. Nobody who has watched a model spill out of memory forgets the experience. The machine is not broken. It is simply out of road.

Unified memory changes the shape

The alternative is a pool that both the processor and the accelerator can address as one. Unified-memory designs remove the copy across the thin link, because there is nothing to copy across. The same physical memory holds the weights, the cache, and the scratch space. The bandwidth is whatever that memory offers, and it can be very high, though it is typically shared with everything else on the package.

This is the shape most hardware for dedicated AI converges on, and for good reason. It lets a machine hold a model larger than any single accelerator's private memory, and it lets the accelerator see all of it at once, which makes long context affordable instead of catastrophic.

The trade-off is that bandwidth is a shared resource. Every byte the model reads is a byte the processor cannot use, and vice versa. On a dedicated machine that trade is easy to manage, because the processor is doing almost nothing. On a general-purpose box, the desktop, the browser, and the model all pull from the same pool, and throughput becomes a matter of who is asking at that instant.

The cache grows with you

There is a second consumer of bandwidth that is easy to forget: the key-value cache. Every token the model has seen produces a key and a value that must be retained so later tokens can attend to it. The cache grows linearly with context length, and its size depends on the number of layers, the number of attention heads, and the precision used to store it.

At short prompts the cache is trivial. At long conversations it can rival the weights themselves. This is why a machine that feels fast at the start of a session can feel slow twenty minutes in, and why context length is a sizing decision and not only a feature. If you plan to run long documents through a local model, size memory for the cache you intend to keep, not for the weights alone.

What quantization actually moves

Quantization lowers the number of bits used to store each weight, which shrinks the model and therefore reduces the bytes that must be read per token. Both effects help, and the second effect is the one that shows up in tokens per second.

The gains are not linear and not free. Moving from sixteen bits to eight roughly halves the footprint and often increases throughput by a similar proportion, with quality loss that is usually small and measurable. Moving to four bits shrinks further but introduces a real quality tax on reasoning tasks, and on some models it produces a confident, well-formed, wrong answer that is harder to spot than an obvious failure.

The practical rule: pick the lowest precision whose evaluation results you have actually checked on your own prompts, and expect the throughput gain to come mostly from the smaller read per token rather than from faster arithmetic. Anyone promising large speedups at the same precision is describing a different bottleneck than the one you have.

Numbers worth checking

When you compare machines, ignore the teraflops headline and collect five figures instead.

Review sites that measure sustained throughput rather than peak specification are worth reading closely. Publications such as ServeTheHome hardware coverage are useful precisely because they run boxes for long periods and report what happens to clocks and temperatures, which is where the difference between a demo and an appliance shows up.

Edge silicon, honestly

The embedded accelerators in the Jetson family are a fair illustration of the bandwidth argument. They do not win on arithmetic. They win because they put a wide memory path next to modest compute and wrap the whole thing in a fifteen-watt envelope, which makes an always-on appliance possible in a way a desktop card is not.

They are also constrained. Capacity is fixed at purchase and cannot be expanded. Bandwidth is shared with the processor. Very large models simply do not fit, no matter how patient you are. For Jetson for dedicated AI work, the honest recommendation is to choose the model after choosing the memory, not the other way around.

Measure your own machine

Specifications predict, measurement decides. The cheapest useful test is to run the model you intend to use, at the quantization you intend to ship, with a prompt of realistic length, and log three things: tokens per second after the first hundred tokens, time to first token, and memory in use. Repeat it while the machine also does what it normally does.

Do that twice and you will know more about your bottleneck than any specification sheet can tell you. If throughput falls apart when the model exceeds device memory, the bottleneck is the link. If throughput is steady but modest and bandwidth is the only difference from a faster machine, the bottleneck is the pool. If throughput is fine but everything stutters when you open a browser, the bottleneck is contention, and the fix is dedication.

Write down the three numbers. Guesswork about hardware is expensive and completely avoidable.

Where this leaves the argument

Bandwidth is a physical property of the machine and it does not negotiate. That single fact explains why a sixteen-core desktop loses to a smaller box with a wider memory path, why adding accelerators does not scale the way people hope, and why the difference between a machine that is technically capable and one that is genuinely pleasant is almost always measured in bytes per second.

It also explains the value of dedication. Give one workload the whole memory system, remove everything that competes for it, and the same silicon produces a different experience. Not a faster one, necessarily. A predictable one, which after a few months of use is the property people care about.

Continue with idle power and the always-on AI box, or return to the index for the full argument.