Dedicated AI Hardware
A machine with one job. It runs models. It does nothing else, and that is the entire point.
Most people meet a language model on a general-purpose computer. It is a laptop that also does email, or a gaming tower that also renders video, or a rented slice of somebody else's cluster. The model is a guest in a house built for something else. It works, in the way that sleeping on a couch works.
Dedicated AI hardware is the opposite arrangement. One box, sized for one workload, powered on all day, holding the model in memory and answering when asked. When people argue about dedicated AI hardware, they usually argue about chip names and benchmark scores. Those numbers matter less than the shape of the machine. Shape decides whether the thing is pleasant to live with or a constant irritant.
This site is an argument for the dedicated shape. It is not a product pitch. It is a claim about engineering trade-offs, made in blunt terms, followed by an honest list of what the idea cannot do.
One job, not twenty
A general-purpose computer is a compromise between dozens of jobs that share nothing but electricity. A spreadsheet wants bursty single-core speed. A video export wants every core at once for two minutes. A game wants a fast GPU for an hour. A model server wants the opposite of all three: steady, modest, uninterrupted use of memory and accelerator, every hour of every day, forever.
Design for the average of those jobs and you get a machine that is mediocre at all of them. Design for one and you get a machine that is excellent at one and useless for the rest. Dedication is a subtraction, not an addition. You remove ports, slots, displays, spinning disks, RGB lighting, and the assumption that a human is sitting in front of it. What remains is a box that boots, loads weights, and answers.
That subtraction is why an idle AI box can be fanless and why a workstation with the same silicon cannot. It is also why the dedicated machine tends to cost less than people expect. You are not paying for capability you will never use.
A desktop is the wrong shape
The failure is structural, not a matter of tuning.
- Interrupt-driven by design. A desktop operating system assumes a person is present. It wakes cores for indexing, updates, telemetry, and background sync. Every one of those wakes steals bandwidth from the model at unpredictable moments, and unpredictable latency is the one thing a serving machine cannot tolerate.
- Cooling tuned for bursts. A desktop cools for short peaks and then idles. A server runs at a middling load for years. Different thermal curve, different fan strategy, different expected life.
- Power states that fight you. Deep sleep, aggressive C-states, and USB selective suspend all save watts on a laptop and all hurt a machine that must answer in under a second, at 3 AM, with no warning.
- Memory and storage shared with everything else. On a general-purpose box, the page cache, the browser, and the model compete for the same bus. On a dedicated box, the model owns it.
None of this means a desktop cannot run a model. It means the desktop is running two jobs at once and pretending otherwise. For a demo that is fine. For something answering every request in a household for three years, it is a slow accumulation of small failures.
Bandwidth beats brawn
Token generation is memory-bound. That sentence is the whole article underneath this one, but it belongs here too because it reframes what you should buy.
Producing one token requires reading the active weights and the key-value cache, roughly once per token. The arithmetic per byte is small. Far small. So the ceiling on tokens per second is set by how fast the machine can move bytes into the compute units, not by how many multiply-accumulate operations those units can perform per cycle.
This is why a modest accelerator with a wide, fast memory path embarrasses a much larger accelerator with a narrow one. It is why unified-memory designs, which let the accelerator see the whole pool at once, behave so differently from designs that shuttle tensors across a thin PCIe link. And it is why quoting teraflops at somebody shopping for a model server is close to meaningless. Ask for bytes per second and for the size of the largest model that fits with room for cache.
Compute determines whether a model can run at all. Bandwidth determines whether it is usable.
Dedicated silicon versus shared silicon
Sharing is the quiet tax on general-purpose machines. When your accelerator is also driving a display, a compositor, and a browser, the model gets a slice, not the whole thing. Latency becomes a distribution instead of a number. Worst-case time to first token can be five times the median, and you will notice it exactly when you are waiting.
Dedicated silicon removes the contention because there is nothing to contend with. The accelerator has no desktop to serve. Its driver stack is minimal. Its scheduler is yours to configure. That is what buyers of ClawBox AI server hardware tend to discover afterwards: the honest win is not peak speed, it is the disappearance of jitter.
There is a second effect. A machine that only runs inference can be tuned once and left alone. Quantization level, context length, batch policy, thread count, power cap. On a shared box these get re-tuned every time something else changes. On a dedicated box they are set on day one and stay set.
Idle power is a design decision
An always-on box has two cost regimes: the watts it burns while thinking and the watts it burns while waiting. Waiting dominates the calendar. A server that draws eighty watts at idle and two hundred under load costs far more per year than its peak number suggests, because peaks are rare and idle is constant.
Dedication lets you attack idle directly. No monitor to drive. No spinning disks to keep awake for no reason. No discrete GPU idling at a floor of thirty watts because the vendor's driver refuses to park it. A well-built single-purpose box can idle in the low tens of watts, sometimes less, and still answer in a second. That is not a marketing figure. It is the arithmetic of removing every component that has no job.
The case against renting inference
Renting inference is convenient and, for a first experiment, correct. Then three things happen.
- Your marginal cost never falls. Every token you ever serve has a price attached. A box you own gets cheaper per token the longer you keep it and the more you run it. The rented line never bends.
- Your capability is somebody else's roadmap. Models get retired. Endpoints get renamed. Prices move. A dependency you did not choose becomes a project you must do anyway, on a deadline you did not set.
- Your data leaves. For many workloads that is fine, and pretending otherwise is dishonest. For a diary, a medical history, a client contract, or a set of family photographs, it is a decision with consequences that compound.
The no-cloud AI philosophy is not paranoia. It is a bet that the value of a private model grows as more of your life becomes machine-readable, and that the cost of a machine you own falls faster than the cost of a subscription you rent.
What dedicated actually buys you
Predictability
Latency you can plan around. The same request takes the same time at noon and at midnight, because nothing else is competing for the machine.
Silence
A box sized for a steady hundred watts can be cooled passively. It can live in a room where people sleep. That changes where it can go.
Longevity
No desktop update cycle forcing a reinstall. No driver churn. A serving appliance that is boring for five years is a success, not a stagnation.
Ownership
The weights are on your disk. The endpoint is on your network. Pull the cable and the thing still answers. That property is worth more than most benchmarks.
Honest limits
Dedication is not a free win, and anyone selling you one is selling.
- A dedicated box is bad at variety. It is tuned for one model family at one quantization at one context length. Ask it to do something else and it will be merely adequate.
- Frontier models still do not fit. The largest open models need more memory than any single household machine has. Dedicated hardware means the largest model you can hold, not the largest model that exists.
- You now own the operations. Backups, failed disks, thermal throttling in August, and the day the power supply dies. Cloud providers absorbed that work; you have taken it back.
- Utilization is low by design. You are paying for capacity that mostly sits idle, in exchange for latency that is always available. If your workload is bursty and rare, renting is genuinely cheaper.
- Hardware ages. Two generations of accelerator is a meaningful gap. A dedicated box is a five-year bet on a fast-moving field, and you should make it with open eyes.
None of that undoes the argument. It bounds it. A dedicated machine is the right answer when the workload is continuous, the data is private, the latency matters, and you intend to keep the thing for years. Outside those conditions, a shared machine is the better engineering choice, and saying so costs nothing.
Where to go next
The rest of this site goes deeper on the three parts of the argument that deserve their own page.
- Memory Bandwidth Is the Real Bottleneck — the physics behind token throughput, and the numbers to check before you buy.
- Idle Power and the Always-On AI Box — what an always-on machine costs, and how to design for the ninety-five percent of hours when nothing is happening.
- The Case Against Renting Inference — a fair hearing on the rent-versus-own question, including the cases where renting wins.
Read them in order if you are new to this. Read article one if you are already shopping and want the only specification that matters.
Once the three parts are settled — memory, thermals and a fixed workload — the remaining choice is whether to build or to buy a finished unit. ClawBox is the built version of this argument: one box, the model already fitted to it, no round trip to a cloud.