Every guide to running a large language model on your own infrastructure opens the same way. Work out the VRAM. Parameters times two bytes, divide by your quantisation, round up to the nearest accelerator you can actually buy. The page quotes a number somewhere between one data-centre GPU and a rack of them, and the conversation with your finance director ends in that paragraph.
I want to put a measurement in front of that paragraph.
On a desktop with an i9-13900K, 64GB of ordinary DDR5 and no graphics card of any kind, a 26-billion-parameter model ran at about 40 tokens per second single-stream and about 124 tokens per second with a handful of requests batched. Nothing rented, nothing in anyone else's data centre, no accelerator in the machine at all.
That number is not interesting as a benchmark. It is interesting because of what it does to a conversation that usually dies early. When a compliance officer says the customer data cannot leave the building, and an architect answers that keeping it in the building means a GPU procurement nobody has budget for, the project stops. The measurement above says the second half of that sentence is not necessarily true.
The short answer
You can run a capable open-weight model on ordinary server or desktop hardware with no GPU. On an i9-13900K with 64GB of DDR5, a 26B mixture-of-experts model served about 40 tokens/sec single-stream and about 124 batched.
The reason the guides get this wrong is that VRAM is not the constraint, memory bandwidth is. VRAM is only a proxy for it. Two things break the proxy: mixture-of-experts models read a small fraction of their weights per token, and batching amortises every byte you read across all the requests in flight.
Self-hosting is rarely the cheaper option and you should not sell it internally as one. It is the option that lets data stay where the law, the contract or the customer says it must stay.
The VRAM question is the wrong question
Look at what the on-premise category actually publishes. The deployment guides that rank for this topic describe the work as sizing: VRAM for the weights, VRAM for the KV cache, VRAM headroom for concurrency. The arithmetic is correct. The framing is what fails, because VRAM was never the thing doing the work.
A GPU is fast at inference for one dominant reason. Generating a token is a memory-bound operation, not a compute-bound one. For each token the machine has to read the weights it needs out of memory and do comparatively little arithmetic with them. High-bandwidth memory on an accelerator moves those bytes at terabytes per second. DDR5 in a desktop moves them at tens of gigabytes per second. That gap, roughly one to two orders of magnitude, is the whole story. VRAM capacity is simply the amount of model you can park next to that fast memory.
Once you see it as a bandwidth problem, the sizing question changes shape. It stops being how many bytes does this model weigh and becomes how many bytes does this model actually read to produce one token. Those are not the same number. For a dense model they are close enough that the distinction does not help you. For a mixture-of-experts model they are wildly different, and that difference is the opening.
What one token actually reads
The model I measured has 128 experts and fires 3 of them per token. So the experts dominate the file on disk and barely participate in any individual forward pass. Meanwhile the attention and embedding machinery, the part that is small on disk, gets read in full on every single token.
The experts are most of the model on disk and the smallest slice per token. The head is the inverse: tiny on disk, read in full every step. Derived from the model file, not estimated.
Read those two bars against each other, because the gap between them is the entire argument. Size the machine by the top bar and you buy hardware for 88% of a model that contributes 16% of the work. Size it by the bottom bar and a different class of hardware becomes viable.
It also tells you where to spend effort. The head is small, so nobody thinks to optimise it, and it is read in full on every token, so it costs more than anything else. I took it from 6.5 bits per weight down to 2.4 and could not find any damage across prompts: 606MB down to 225MB on the single most-read tensor in the model. The full derivation, including the byte accounting, is in the writeup of that build.
Two dead ends are worth recording, because they bound the technique rather than advertise it. At 1.75 bits the head got slower, not faster: more time spent unpacking the tighter format than saved reading it, and the output began looping in the arithmetic. And when I cut from 3 experts to 8, perplexity on raw Wikipedia text jumped by 1.6x, which looked alarming until I remembered that raw-text perplexity is close to meaningless for a chat-tuned model. Reading the actual outputs on real prompts, top-3 and top-8 gave the same answers. I trusted the reading over the metric.
The wall is real, and it is not where you think
None of this means memory bandwidth stops mattering. It means you should measure yours rather than infer it from a spec sheet.
Decode already runs at roughly 78% of what this memory can really deliver. Pinning threads and changing core counts moved none of it.
This is the honest half of the argument. Single-stream generation on this machine is genuinely close to the memory wall. There is no configuration trick left. I tried the usual ones and they moved nothing, because the bottleneck is not scheduling, it is physics.
What moves it is batching. Serve several requests at once and you read each weight once for all of them. The matrix multiply stops being a vector operation and becomes a real matrix operation, the bottleneck migrates from memory to compute, and the cores that were sitting idle finally have work. That is the whole distance between 40 tokens per second and 124.
Which has a direct consequence for how you plan a deployment. A CPU-served model is a poor fit for one impatient human waiting on a chat window, and a good fit for a queue: document processing, extraction pipelines, classification, overnight batch work, an agent doing many small calls. Most enterprise AI work is a queue. The demo is what is interactive.
What sovereignty actually buys you
Now the part the hardware conversation was blocking. If keeping the model inside your own network is achievable on hardware you can requisition without a board paper, what do you actually get for it?
Data that legally cannot leave
This is the only argument that matters in regulated work, and it is not really an argument about technology. Some organisations hold data where sending it to a third-party endpoint is not a risk to be managed but a thing they are not permitted to do: patient records, legal privilege, defence work, personal data under a residency requirement, a client contract with an explicit no-subprocessor clause. No amount of vendor assurance changes the answer, because the control they need is architectural, not contractual. Once the weights are on your disk, inference needs no network at all. An air-gapped model cannot leak to a provider because there is no provider.
A model that is the same next year
This one is consistently undervalued and it is the one I would put second. A hosted model is a moving target. It is updated, deprecated, re-tuned and silently re-routed, and none of those events are under your control or necessarily announced in a way your team will notice. If you have built an evaluation suite, tuned prompts against specific behaviour, or validated a pipeline for an auditor, that validation has a shelf life set by somebody else's release schedule.
A model on your own disk does not drift. The weights are a file. You upgrade when you choose, you can run the old and the new side by side, and you can prove to an auditor that the system which produced last quarter's decisions is byte-identical to the one running today. For anything that has to be defended after the fact, that property is worth more than a few points of benchmark score.
No refusal surface you did not design
Hosted models carry safety behaviour tuned for a global consumer product. That is reasonable of the vendors and frequently wrong for a specific professional context. Insurers processing accident descriptions, clinicians handling clinical language, security teams analysing malware, lawyers working on violent crime: all of them hit refusals that have nothing to do with their actual risk profile, and no configuration flag fixes it. When you serve the model, the only policy in the path is yours.
Availability that does not depend on anyone
No rate limits, no capacity throttling at peak, no dependency on a provider's status page, no outage in a region you do not operate in taking down a process you do. And it works with the internet unplugged, which matters more than it sounds for factory floors, ships, field sites, secure rooms and anywhere else the network is not a given.
The commercial asymmetry
Worth saying plainly because it is the part people feel but rarely write down. The current generation of hosted AI is priced to win a market, not to make a margin. Whatever your contract says today, you are building a dependency on infrastructure you do not control, priced by a company whose pricing power grows precisely as your dependency does. Self-hosting some part of your workload is not a prediction that this goes badly. It is optionality, and optionality is cheap to hold and expensive to acquire in a hurry.
When self-hosting is the wrong call
I would rather talk you out of this than have you do it for the wrong reason, so here is the honest list. Self-hosting is usually the wrong answer when:
You are doing it to save money at low volume. API prices per token have fallen far enough that the arithmetic almost never works for modest usage, and the running costs people forget, engineering time above all, dwarf the token bill. If cost is the only driver, stop here.
You need frontier reasoning. For genuinely hard open-ended reasoning, the best hosted models are still ahead of anything you can serve yourself, and it is not close. Self-host the scoped work and route the hard tail to an API if your data rules allow it. That hybrid is the right default for most organisations.
One person is waiting on the answer. Interactive, latency-sensitive, single-stream chat is the workload CPU inference is least suited to. Batch work is where it shines.
Nobody owns it after launch. A self-hosted model is infrastructure. It needs patching, monitoring, capacity planning and somebody who understands why it got slower. If no team owns that, you are building an outage with a delayed fuse.
The questions worth asking, in order
If you are the person who has been handed "look into running this ourselves", these are the four questions that decide it, and the order matters more than any individual answer.
- Is there data here that cannot leave? Not would rather not leave. Cannot, because of law, contract or residency. If yes, the rest of this is an implementation problem and you should proceed. If no, be sceptical of the whole idea, because you are probably about to rebuild something you could rent.
- Is the workload a queue or a conversation? Queues batch, and batching is what makes commodity hardware viable. A conversation cannot be batched and needs the low single-stream latency an accelerator gives you.
- How good does the model actually have to be? Scope it honestly against your real task, not a benchmark. A great deal of production AI work is extraction, classification, routing, summarisation and tool calling, where a well-chosen open-weight model in the 20B to 70B range is genuinely hard to distinguish from a frontier one. Test it on your own data before you accept anyone's ranking, mine included.
- Who runs it in eighteen months? If you cannot name them, solve that before you buy hardware.
How to size the machine
If the answers point to self-hosting, size it in this order, which is close to the inverse of how the guides do it.
Memory bandwidth first. This sets your ceiling on tokens per second, and it is the number you should benchmark on the actual machine rather than read off a spec sheet. My DDR5-4800 advertises 76.8 GB/s and delivers 64.5. Assume a similar haircut and verify it.
Capacity second. Enough RAM to hold the quantised weights plus the KV cache for your target concurrency, with room to spare. Capacity is what decides whether a model runs at all; bandwidth decides how fast. People buy these in the wrong order.
Architecture third. A mixture-of-experts model at a given parameter count reads far fewer bytes per token than a dense model of the same size. If you are CPU-serving, that difference is the difference between viable and not. This is the single highest-leverage choice on the list and it is usually made last, if at all.
Cores last. They matter for prompt processing and for batched decode, and they are close to irrelevant for single-stream generation. Buying cores to fix a bandwidth problem is the most common way to waste money here.
What I would actually do
Start with the smallest honest pilot. Take one workload that is already a queue, where the data is already sensitive, and where you can measure the output quality against something. Serve an open-weight MoE model on a machine you already own. Measure your real bandwidth, measure tokens per second under your real batch size, and evaluate the outputs on your own documents rather than a public benchmark.
You will learn more in a fortnight than any amount of vendor sizing advice will tell you, including mine. And if the answer comes back that you need an accelerator after all, you will know that from your numbers rather than from a formula on a marketing page.
The thing I would not do is let the VRAM paragraph end the conversation before anyone has measured anything. That paragraph has quietly talked a lot of organisations out of keeping their own data, and as far as I can tell it is answering a question about capacity when the real question was always about bandwidth.
Common questions
Can you run an LLM without a GPU?
Yes. On an i9-13900K with 64GB of DDR5 and no graphics card, a 26-billion-parameter mixture-of-experts model ran at roughly 40 tokens per second single-stream and roughly 124 with requests batched. The constraint on CPU inference is memory bandwidth rather than the absence of an accelerator, so what matters is how many bytes your model reads per token and how fast your memory can move them.
What hardware do you need for an on-premise LLM?
Size memory bandwidth first, RAM capacity second, model architecture third and core count last. VRAM figures quoted in deployment guides are a proxy for bandwidth, and the proxy breaks for sparse models and for batched serving. Benchmark the bandwidth on the actual machine, because the achievable figure runs meaningfully below the advertised one.
Is a self-hosted LLM as good as ChatGPT or Claude?
Not at the frontier, and anyone telling you otherwise is selling something. For open-ended reasoning the hosted frontier models remain clearly ahead. For scoped production work, classification, extraction, summarisation, routing and tool calling, a well-chosen open-weight model in the 20B to 70B range is frequently indistinguishable in output while running entirely inside your network. Evaluate on your own data rather than on a leaderboard.
Is self-hosting an LLM worth it?
It is worth it when your data cannot legally or contractually leave your network, when you need the model to behave identically in a year's time, or when it has to work without an internet connection. It is rarely worth it purely to reduce spend at modest volumes, because per-token API pricing has fallen far enough that engineering time dominates the comparison.
What is the difference between a self-hosted, on-premise, private and local LLM?
Mostly marketing, and the overlap causes real confusion in procurement. A local LLM usually means a single workstation. An on premise LLM means inside an organisation's own network boundary. A self hosted LLM covers any deployment where you control the serving infrastructure, including your own cloud tenancy. Private LLM is the loosest of the four and the one to watch, because vendors apply it both to genuinely self-hosted models and to hosted endpoints with a confidentiality agreement attached. Those are not the same guarantee. The property that actually matters for data governance is whether a third-party API sits in the request path, so ask that question rather than accepting the label.
Can a self-hosted LLM run air-gapped?
Yes. Once the weights are on disk, inference requires no network access whatsoever. This is why self-hosting is viable for classified, regulated and offline environments, and it is a guarantee no hosted API can offer at any price, because it is a property of the architecture rather than of the contract.
Does a mixture-of-experts model really help on CPU?
Substantially, and it is the most underrated choice in the whole exercise. A MoE model activates only a few of its experts per token, so it reads a small fraction of its total weights on each forward pass. In the model measured here the experts are roughly 88% of the file on disk but only about 16% of the bytes read per token. Since CPU inference is bandwidth-bound, reading fewer bytes translates almost directly into more tokens per second.
Read next
Running Gemma-4 26B at 124 tokens/sec on a CPU, no GPUThe full build behind the numbers in this piece: the byte budget, the quantisation recipe, and the dead ends.