The Always-On AI Server
A machine that is awake all year spends most of those hours doing nothing. The nothing is where the money goes.
Every conversation about an always-on AI server starts with speed and ends, a few months later, with the electricity bill. That is not because thinking is expensive. It is because thinking is rare. A household assistant sees perhaps a few dozen requests a day, each lasting seconds. The remaining twenty-three and a half hours are standby, and standby is not free.
This page is about designing and choosing for the standby regime: the idle power floor, why general-purpose machines are bad at it, and which decisions actually move the number.
What "always on" really means
Always on does not mean always busy. It means always answerable. The machine must hold the model in memory so that the first request of the morning does not wait four minutes for weights to load from disk. It must keep its network stack live so the request can arrive. It must keep its thermal and power systems stable so that waking up is instant rather than a negotiation.
Those obligations set a floor. Something must remain powered, something must remain in RAM, and something must remain listening. The engineering question is how low that floor can go while the machine stays instantly available. Good appliance designs push it to the low tens of watts. Careless ones sit above a hundred, permanently, for a workload that fires for a few seconds at a time.
An idle draw of one hundred watts for one year is roughly the same energy as running a small refrigerator. Nobody buys a refrigerator to answer questions twice an hour.
The idle floor is a sum of parts
Idle power is not a property of the processor. It is the total of everything that stays awake.
- The processor package, which has a minimum draw below which it cannot go without losing the ability to respond quickly.
- The accelerator, whose idle floor is set by the vendor driver. Some park gracefully. Some do not, and hold tens of watts indefinitely for no benefit.
- Memory, which must be refreshed continuously, and which grows with the size of the model you keep resident. More capacity means a higher floor, always.
- Storage, where a spinning disk that never parks can cost several watts on its own, and where any drive that must stay ready to serve adds a little more.
- The network interface, which mostly sits in an efficient state but wakes for every broadcast on the network, and can be poked awake constantly on a busy segment.
- Fans and voltage conversion, where a cheap power supply can waste ten to fifteen percent of everything the machine consumes, awake or asleep.
Add these honestly and the floor is visible. Subtract the ones that have no job and the floor drops. Dedication is largely the discipline of subtracting.
Why general-purpose machines cannot idle well
A desktop is optimized for a person sitting in front of it. That assumption leaks into every power decision.
The display subsystem stays ready in case a monitor appears. The audio subsystem stays alive in case something wants to play. Background services phone home. The file indexer wakes on a schedule you did not set. The updater decides at 4 AM that now is the moment. Each of these individually costs fractions of a watt. Together they keep a machine from ever reaching its true floor, and worse, they inject unpredictable latency into the moments when a request does arrive.
There is also the sleep problem. A desktop saves power by suspending. A suspended machine cannot answer. Systems that wake on network traffic exist, but they add seconds to the response and they fail in interesting ways after firmware updates. For an appliance, the correct configuration is usually to disable most sleep states entirely and instead design a low enough idle floor that sleeping is unnecessary. That is only possible on a machine whose components were chosen for it.
Designing for the ninety-five percent
If you are building rather than buying, five decisions set most of the idle number.
- Pick a platform with a low package floor. Embedded and small-form-factor boards are built around a power budget from the start. Desktop parts are not, and their minimum idle draw is correspondingly higher.
- Prefer unified memory. One pool means no discrete device sitting at a vendor-imposed idle floor waiting for work that is not coming.
- Use solid-state storage only. No spindles, no parking timers, no vibration.
- Buy an efficient power supply sized close to the real load, not to a fantasy peak. An oversized supply is inefficient at the very point where the machine spends its life.
- Remove the desktop stack. A minimal system with no graphical session has nothing to keep awake. This is where a stripped-down appliance operating system earns its keep, and where selfhost AI builds differ most visibly from repurposed desktops.
Do all five and the idle floor can fall below the point where it is a rounding error in a household budget. Do none and you have built a space heater with a language model attached.
Thermals follow power
Idle power and idle heat are the same quantity. A box that draws fifteen watts at rest can be cooled by a heatsink with no moving parts, which means it makes no noise and cannot fail by fan seizure. A box that draws ninety watts at rest needs airflow, which means a fan, which means noise and a wear item.
That matters beyond comfort. A fanless machine can live in a bedroom, a kitchen, or a workshop shelf. A machine with a constant whine gets banished to a cupboard, and a cupboard is where thermal throttling lives. Placement is a power decision wearing a different hat.
Do the arithmetic before you build
Estimate the annual energy cost of the idle floor, then compare it with what you would pay to rent the same capability for a year. Then look at what you are actually running. Most personal AI workloads are bursty, low-volume, and latency-sensitive, which is the worst possible match for a high idle floor and the best possible match for an efficient appliance.
The numbers are easy. Watts times hours in a year, divided by a thousand, times your local price per kilowatt-hour. Do it for seventy watts and for fifteen. The gap is usually more than the difference in purchase price between a repurposed desktop and a personal AI server designed around a small power budget.
Two honest caveats. First, hardware ages and efficiency improves, so a five-year projection will overstate the cost of the later years unless you plan to refresh. Second, local electricity prices vary by a factor of five between regions, which changes the break-even point considerably. Check your own tariff rather than trusting a general figure.
When always-on is the wrong choice
There are workloads where a permanently resident model is simply not worth the standby cost.
- Occasional use. If you touch the model twice a week, a rented endpoint costs less and asks nothing of you.
- Enormous models. Holding a frontier-scale model resident needs memory whose idle refresh cost is real, and the machine that could do it will never idle cheaply.
- Bursty batch work. Processing a thousand documents once a month is a job for a rented machine that exists for an hour, not an appliance that exists for a year.
Always-on is for continuous, private, latency-sensitive use with modest volume. That describes a household assistant, a family archive, a small business knowledge base, or a developer's local coding companion. It does not describe everything, and pretending otherwise is how people end up with an expensive box they unplug.
The bottom line
Idle power is the specification nobody advertises and the one that decides whether a machine is a pleasure or a monthly irritation. Buy for the standby regime, because that is where the machine lives. Then check the peak numbers, because those are the ones you will actually notice.
Next, read the case against renting inference for the full cost comparison, or go back to the bandwidth article if you still need to size memory.