- An on-premises LLM runs on hardware you control, so prompts, documents and answers never go to a provider.
- It takes four layers: GPU hardware, model weights you hold, serving software such as vLLM, and the application people use.
- Running a model yourself trades per-token API bills for upfront hardware and the work of operating it.
- Small language models fine-tuned for one job are a common choice, because they fit on modest hardware.
What is an on-premises LLM?
An on-premises LLM is a large language model you run yourself, on hardware you control, rather than one you reach over the internet through a provider's API. Prompts, documents and answers never leave your environment, and nobody else can change, reprice or retire the model you depend on.
People also call it an on-prem or on-premise LLM. Two neighbors get mixed in: "self-hosted" is broader, since it includes running the model in your own cloud account, and "private LLM" is a vendor label with no fixed meaning.
How an on-premises LLM works
Four layers do the work, all inside your environment.
| Layer | What it is | Examples |
|---|---|---|
| Hardware | GPU servers in your data center, or a private cluster | GPU servers, a Kubernetes cluster |
| Model | Weights you download or fine-tune, stored locally | Open-weight models such as Llama, Mistral or H2O Danube |
| Serving | Software that loads the model and answers requests | vLLM, Hugging Face Text Generation Inference |
| Application | What people use | A chat assistant, document Q&A, agents |
The serving layer is where most of the engineering sits. vLLM, a common open-source choice, describes itself as "a fast and easy-to-use library for LLM inference and serving." On top, most applications use retrieval-augmented generation so the model answers from your own documents. The model doesn't need the internet to answer: if the servers have no outside connection at all, the LLM is also air-gapped.
On-premises LLM vs cloud LLM API
| On-premises LLM | Cloud LLM API | |
|---|---|---|
| Where prompts and data go | Stay on your infrastructure | Sent to the provider |
| How you pay | Hardware and staff up front, then capacity you own | Per token, rising with use |
| Which models | Open-weight and fine-tuned models you hold | The provider's models, which it can change or retire |
| Who decides on updates | You | The provider |
| Who runs it | You | The provider |
Why run an LLM on-premises
- Data control. Regulated and confidential data never goes to a third party.
- Cost at scale. Per-token bills grow with use; owned capacity doesn't. A 2025 cost-benefit analysis estimates the point where running open models locally breaks even with commercial LLM services, and finds it depends on usage levels and the performance you need.
- Customization. You can fine-tune a model on your own data and keep the result.
- Stability. A model you hold doesn't change under you, so an application that works today keeps working.
Challenges of running an LLM on-premises
- Upfront hardware. GPUs are expensive, and capacity is fixed once installed.
- Model size. The largest models need far more hardware than most data centers have, and many frontier models aren't available to download at all.
- Skills. Serving, scaling and monitoring models is real engineering work.
- Keeping current. New models and patches are yours to evaluate and install.
Small language models on-premises
Many organizations run small language models on-premises: models with a few billion parameters, often fine-tuned for one job, that fit on modest hardware and answer fast. A large North American telecom operator cut its token costs 10x by moving to a small language model on its own hardware.
On-premises LLM examples
- A federal research agency. NIH runs Enterprise h2oGPTe fully air-gapped for more than 8,000 employees across 28 institutes.
- Banks and insurers. Analysts ask questions of internal policies, filings and claims without sending them to an outside API.
- Manufacturers. Plants run assistants on factory networks that are kept separate from the internet.
On-premises LLMs in H2O AI
H2O AI Hybrid Cloud runs the full H2O AI Cloud on your infrastructure, on-premises or fully air-gapped, with any LLM you choose. Enterprise h2oGPTe answers from your documents with citations. H2O Cascade fine-tunes and distills small models on your own data. The open-weight H2O Danube, H2O Mississippi and H2O Lightning models run on your hardware, and H2O AI Super Agent routes each job to the right model.