Most conversations about a self-hosted LLM start with the wrong question. Somebody in the business says the customer data must not leave the company's own systems. Somebody in engineering answers that keeping it there means buying GPUs nobody has budget for, and the project stops there. Both of them are treating it as a hardware question. The real question is where the data goes: which data leaves, who receives it, and what you have to run yourself so that it does not. I have run open models on my own hardware, measured them, and set up the interfaces that let a system be built and tested before the hardware arrives. This is what I would tell the person who has been handed "look into running this ourselves".
The short answer
A self-hosted LLM means the model weights run on infrastructure you control, so prompts, documents, completions and logs never pass through systems you do not control. It is rarely the cheaper option, and you should not present it inside the company as one.
Choose it when the law, a contract or a residency rule says the data cannot leave; when the model must behave identically next year; or when it has to work with the internet unplugged. There are four ways to get there, and for work that can wait in a queue one of them needs no graphics card at all.
What "does not leave the company" means
When a team uses a hosted model, more data leaves the company than the prompt. Write the list down, because a self-hosting plan fails on any item it misses.
- The prompt, which in real work contains the document, the customer record or the contract you are asking about.
- The completion, which contains the model's reading of it.
- The provider's logs, kept for whatever period their terms say, on whatever retention setting your account has.
- Embeddings, if you send text to a hosted embedding model or store vectors in a hosted database. Vectors are derived from the text and can be treated as the text for governance purposes.
- Telemetry from client libraries and observability tools, which often carry prompt fragments in traces.
- Evaluation runs, which are the same data sent again, usually by a different team with a different account.
- The fine-tuning set, if you fine-tune a hosted model, which is the most sensitive data you have, handed over in bulk.
A "private" endpoint with a confidentiality clause does not change where any of those bytes go. It changes who promised what about them. For some data that is enough. For patient records, legal privilege, defense work, personal data under a residency requirement, or a client contract with a no-subprocessor clause, sending it out is not permitted at all, and the protection has to come from how the system is built, not from a contract.
Four ways to keep the data in-house
They range from least to most control, and the useful comparison is where the data goes, what it costs to run, and who has to run it.
| Option | Where the data goes | Model choice | Cost shape | Who runs it | Fits when |
|---|---|---|---|---|---|
| API with a zero-retention contract | To the provider, processed, not stored. You are trusting a contract and an audit. | Frontier models | Per token, no capital | The provider | The data is sensitive but permitted to leave under contract, and you need frontier reasoning. |
| Private endpoint in your own cloud tenancy | Stays inside your cloud account and region. The provider's model, your boundary. | Whatever the cloud offers, open weights included | Per hour for the instance, plus per token on some | Your cloud team | Residency rules are about region and tenancy, not about the public internet. |
| Your own server, CPU only | Never leaves your network. Works air-gapped. | Open-weight models, mixture-of-experts strongly preferred | Hardware you may already own, plus people | You | The workload is a queue: documents, extraction, classification, overnight batches, an agent making many small calls. |
| Your own server, GPU | Never leaves your network. Works air-gapped. | Open-weight models, dense or sparse | Accelerator procurement, power, plus people | You | People are waiting on the answer, or volume is high enough that throughput per watt decides it. |
Most organizations that need this end up with two of the four: their own hardware for well-defined tasks on the data that cannot leave, and a zero-retention API for the hardest cases where the data rules allow it. That hybrid is the right default, and deciding which work goes on which side is the first real design decision.
Size by the workload, not by the model
The sizing guides start from the model: parameters times bytes, divide by quantization, round up to an accelerator. Start from the workload instead. Is it a queue or a conversation? A queue can be batched, and batching is what makes ordinary hardware viable, because every weight read from memory serves all the requests in flight. A conversation cannot be batched, and one person waiting on a chat window needs the low single-stream latency an accelerator gives.
I measured the CPU option rather than assuming it. On a desktop with no graphics card, a 26-billion-parameter mixture-of-experts model ran at about 40 tokens per second for a single stream and about 124 tokens per second with a handful of requests batched. The constraint was memory bandwidth, not the missing accelerator, and the choice of a sparse model made the biggest difference. The full build, the byte accounting and the failed attempts are in the Gemma-4 write-up; the point for this page is that most enterprise AI work is a queue, and usually only the demo is interactive. Size for the queue.
Then work out tokens per day. Count the documents, multiply by their length plus the answer, add the retries, and you have a daily token budget. Divide by the seconds you are willing to spend and you have the throughput you need. Compare that to what a machine you already own measures, not to a spec sheet, because achievable bandwidth is noticeably lower than advertised.
Which model, and checking the license
For well-defined production tasks, extraction, classification, routing, summarization and tool calling, a well-chosen open-weight model between 20 and 70 billion parameters is hard to tell from a frontier model on your own documents. That is the test that matters: your documents, your task, your reviewer, not a public leaderboard. Run the candidates on a few hundred real cases with the answers known and count.
Two things get skipped. The weight license: open weights are not all open for commercial use, and some carry terms about what you may build. Read it before the pilot, not after. And the file itself: pin the exact model file and its hash. The model you validated for the auditor must be byte-identical to the model that ran last quarter's decisions, and with the weights on your disk you can prove that. A hosted model changes: the provider updates and reroutes it on their own schedule, so any evaluation you ran against it can go out of date whenever they decide.
What it costs to run after day one
This is the part that is missing from the hardware conversation, and it is where self-hosting stops being cheaper. A self-hosted model is infrastructure, and infrastructure needs an owner.
- Patching the serving software, the operating system and the model runtime, each on its own schedule.
- Monitoring tokens per second, queue depth and memory, so you know why it got slower before the people waiting on it do.
- Capacity planning as usage grows, which it will, because once a model costs a team nothing per request, every team finds a use for it.
- Evaluation re-run on every model change, against your own cases, with the results kept.
- Input and output checks. Prompt-injection detection on every input, output checks against the source, and a stuck-loop detector for anything agentic. I built a pay-per-call firewall for exactly those three failure modes, and the reason it exists is that a self-hosted model has no vendor safety layer in front of it. The only safety rules on the inputs and outputs are yours, so any gap in them is yours too.
- A named owner for all of the above, before the hardware order, not after.
Against that, the token bill from an API at modest volume is small. Self-hosting is not a way to save money. It is a way to keep data where it must stay, and it is worth paying for when that is the requirement.
What you get for it
Four things, in the order I would rank them.
- Data that cannot leave stays. Once the weights are on your disk, inference needs no network. An air-gapped deployment is possible, and no hosted service can offer that at any price.
- A model that is the same next year. You upgrade when you choose, run old and new side by side, and prove to an auditor what produced a given decision.
- No refusals you did not choose. Hosted models carry safety tuning for a global consumer product. Insurers processing accident descriptions, clinicians handling clinical language, security teams analyzing malware, lawyers working on violent crime all hit refusals that have nothing to do with their risk, and no configuration flag fixes it.
- Availability that does not depend on an outside provider. No rate limits, no throttling at peak, no status page. Factory floors, ships, field sites and secure rooms do not have reliable internet, and the model still works.
When self-hosting is the wrong choice
- To save money at low volume. The arithmetic almost never works, and engineering time costs far more than the token bill. If cost is the only driver, stop here and get a zero-retention contract.
- For frontier reasoning. On hard, open-ended reasoning the best hosted models are still ahead, by a wide margin. Self-host the well-defined tasks and send the hardest cases to an API if your data rules allow it.
- For one person waiting on a chat window. Interactive, single-stream chat is the workload CPU inference suits least. If that is the job, it is a GPU job.
- When nobody owns it afterwards. If you cannot name the team, solve that before you buy anything.
- When "cannot leave" really means "would rather not". Be honest about which one it is. The first is an implementation problem. The second is usually a contract negotiation.
How I would set it up
The same way I run any engagement inside a company. Week one is a decisions file: which data cannot leave and under what rule, which workload goes first, which of the four options it runs on, and who owns it in eighteen months. Then the smallest pilot that gives a real answer: one workload that is already a queue, on data that is already sensitive, with output you can measure against known answers, served on a machine you already own. Every outside service, the document store, the vector index, the queue, sits behind an interface with both a real implementation and a mock, so the whole pipeline runs and is tested before any hardware arrives or any credential is issued. Measure real bandwidth, real tokens per second at your real batch size, and quality on your own documents. You will know in two weeks whether the CPU box is enough or an accelerator is needed, from your numbers rather than a formula on a marketing page. Then the boundary checks, the evaluation suite, the monitoring, and a written handover to the team that owns it. The how it runs page has the week-by-week mechanics, and the page for business owners explains what it asks of you.
Common questions
What is a self-hosted LLM?
A model whose weights run on infrastructure you control, so prompts, documents, completions and logs never pass through systems you do not control. A server in your building, a machine in your own cloud tenancy, or an air-gapped box. What defines it is where the data goes, not the hardware.
What is the difference between a self-hosted, on-premise, private and local LLM?
A local LLM usually means one workstation. On-premise means inside your own network boundary. Self-hosted covers any deployment where you control the serving infrastructure, including your own cloud tenancy. Private is the loosest term and the one to check in procurement, because vendors use it for both truly self-hosted models and hosted endpoints with a confidentiality clause. Those are different guarantees.
Do you need a GPU to self-host an LLM?
Not for work that can wait in a queue. A 26-billion-parameter sparse model ran at about 40 tokens per second single-stream and about 124 batched on a desktop with no graphics card. One person waiting on a chat window is the workload a CPU box suits least. Documents, extraction, classification and overnight batches are where it fits.
Is a self-hosted LLM cheaper than an API?
Rarely. At modest volume the engineering and operating time dominates the token bill. Self-host because the data cannot leave, because the model must be identical next year, or because it must work offline. If cost is the only reason, get a zero-retention contract instead.
Which model should you self-host?
For extraction, classification, routing, summarization and tool calling, an open-weight model between 20 and 70 billion parameters is hard to tell from a frontier model on your own documents. Evaluate on your data, check the weight license allows your use, and pin the exact file so the model you validated is the one that runs.
Can a self-hosted LLM run air-gapped?
Yes. Once the weights are on disk, inference needs no network at all. It is a property of the architecture, not of any contract.
Who should own it afterwards?
A named team, before any hardware is bought. Patching, monitoring, capacity, evaluations on every model change, and prompt-injection checks on inputs and outputs. If nobody owns that, the system will eventually fail and nobody will be responsible for fixing it.
Agentic AI architect. Twenty years in enterprise architecture, seven companies built and sold, more than 8,000 commits in the last twelve months across a retail platform, an ad-acquisition platform, a transactional email service, a video product, robotics and CPU inference research. Every number on this site was measured on real hardware or taken from a real engagement's records.
Working with me
I take on one or two engagements at a time, inside the company, and I hand over when it runs. If you have a problem in your own processes that no single product solves, and you need it built and running, not just a recommendation, that is the kind of work I do.
The measurement
Running Gemma-4 26B at 124 tokens/sec on a CPU, no GPUThe build behind the CPU option: the byte budget, the quantization recipe, the roofline, and the failed attempts.
The role
What is a forward deployed engineer?The engagement model this page's setup section describes, from someone who does it.