Local LLM On-Premise: How to Run Private AI Inside Your Company, and When a Hosted API Is the Better Choice
A local LLM on-premise is a language model whose weights you download and run on servers you control, in your own data centre or your own cloud account. Prompts, documents and answers stay inside your network. This is what most people mean by private AI or a self-hosted LLM for business. It is the right choice when data may not leave the company, when a contract or regulator says so, when steady volume makes cost predictable, or when a site has no reliable internet.
It is not the right choice for everything. A hosted API gives you the strongest models with no hardware and no operations work, and for many workloads that is the better deal. This guide covers the decision, the models and tools, hardware sizing and safe operation.
When a local LLM on-premise makes sense
- Data may not leave the network. Design files, source code, customer records, production data. If the data cannot go to a third party, the model has to come to the data.
- Regulation or contracts. Sector rules, customer agreements or internal policy can forbid external processing. On data protection, see GDPR and AI integration.
- Predictable cost at steady volume. A hosted API bills per token. A local model costs the same whether it answers one request or works all day.
- Offline or factory environments. On an isolated plant network, a model that needs the internet stops when the link does.
When a hosted API is the better choice
- You need the strongest models. The frontier models from providers such as Anthropic and OpenAI are not published as weights.
- Volume is low or uneven. Idle hardware is expensive per answer.
- You are still testing the idea. Prove the use case with an API on non-sensitive data first.
- Nobody can operate it. A model server needs an owner.
Many companies use both: a hosted model for general work and a local one for the data that must stay inside. Our Claude API integration guide covers the hosted side.
Hosted API vs local model at a glance
| Question | Hosted API | Local model |
|---|---|---|
| Where is data processed? | At the provider, under a contract | On your servers, inside your network |
| Cost shape | Per token, grows with use | Hardware and setup first, then flat |
| Model quality | Largest frontier models | Open-weight models, often enough for defined tasks |
| Who operates it? | The provider | You or a partner |
| Works without internet? | No | Yes |
Open-weight models: Qwen, Llama, Mistral and what the licence allows
Open-weight means the trained parameters are published for download, so you can run the model yourself. It is not the same as open source: the Open Source Initiative's definition of open source AI also asks for the training code and information about the training data. In practice the licence decides what you may do.
- Qwen (Alibaba Cloud): the project repository states that its open-weight models are under Apache 2.0, a permissive licence that allows commercial use.
- Llama (Meta): Meta's own community licence. Commercial use is allowed, with an acceptable use policy and attribution rules if you redistribute the model.
- Mistral (Mistral AI): some models are Apache 2.0, others under Mistral's own terms.
- Gemma (Google): open weights under terms that differ by generation.
Read the licence file of the exact model and version before you build on it. Then choose by testing your own tasks in your own language, not by a leaderboard.
Hardware sizing for private AI: memory, quantization, concurrency
GPU memory is the main constraint. Three things use it:
- The weights. Roughly parameter count times bits per weight. At 16-bit precision, a billion parameters take about 2 GB.
- The context. Every request holds working memory that grows with the length of prompt and answer.
- Concurrency. Each simultaneous request needs its own working memory.
Quantization stores weights at lower precision. A 4-bit version needs roughly a quarter to a third of the 16-bit memory, at some cost in answer quality, so test it on your own questions. Models can also run on CPUs, which is slower but can be enough for batch jobs.
Size for the peak number of simultaneous requests, not for the number of employees. Buy hardware after a pilot has measured this, not before.
Serving stack: vLLM, Ollama, llama.cpp and TGI
- vLLM: an open-source serving engine that started at UC Berkeley. It batches incoming requests continuously and suits many concurrent users on GPUs.
- Ollama: an open-source tool that runs models with one command and exposes a local API. Good for a pilot or a small team.
- llama.cpp: a C/C++ engine that runs quantized models in the GGUF format on CPUs and GPUs. Useful on modest hardware.
- Hugging Face TGI: a serving toolkit found in many existing setups. Its documentation now says it is in maintenance mode and recommends engines such as vLLM or llama.cpp.
vLLM and llama.cpp both offer an OpenAI-compatible endpoint, so applications can switch between a hosted and a local model with little change.
Connecting a self-hosted LLM to company data
The model knows nothing about your company. The usual pattern is retrieval: documents are split into passages and indexed in a vector database such as pgvector for PostgreSQL or Qdrant. A question retrieves the relevant passages, and the model answers from them and names its sources.
Permissions are the part that goes wrong. Retrieval must filter by the rights of the person asking, or the assistant becomes a way around your file permissions. For structured systems, use their APIs with read-only accounts, as described in ERP and AI integration.
Operating a local model
- Monitoring: GPU memory, queue length, response time, errors.
- Updates: model versions, serving software and drivers. Test each change against a fixed set of questions.
- Capacity: watch peak load and plan the next hardware step early.
- Backups: configuration, prompts, the document index, a verified copy of the weights.
- Access control: single sign-on for people, separate keys for applications.
- Audit logs: who asked what and which documents were retrieved, with a retention period.
Security: isolation, no outbound calls, model supply chain
- Network isolation. Inference servers sit in their own segment, reachable only through an authenticated gateway.
- No outbound calls. Block egress from the model servers and bring updates in through a controlled mirror.
- Model supply chain. Weights are files from the internet. Hugging Face warns that pickle-based model files can execute code when loaded and that its scanning is no guarantee. Download from the publisher's official account, prefer the safetensors or GGUF formats, pin the revision, record checksums and keep an internal copy.
- Retrieved text is data. Documents can contain instructions aimed at the model, so limit what it is allowed to trigger.
A staged plan
- 1. One use case. Collect real questions with known good answers.
- 2. Classify the data. What may leave the network, what may not.
- 3. Pilot. Compare two or three open-weight models on small or rented hardware.
- 4. Measure. Answer quality, response time, peak concurrency.
- 5. Build production. Serving, gateway, retrieval with permissions, monitoring, logs.
- 6. Roll out. One team first, then wider.
The organisational side is covered in how to integrate AI into your company.
Common mistakes
- Buying GPUs before testing a model.
- Choosing the model by benchmark rank.
- Not reading the licence.
- Indexing file shares without their permissions.
- Leaving outbound access open on model servers.
- Going live with no owner for operations.
Checklist before you start
- Use case and test questions defined
- Data classified: what must stay inside
- Model licence read and approved
- Peak concurrent requests measured
- Retrieval respects user permissions
- Egress blocked, weights verified and mirrored
- Monitoring, backups, audit logs and a named owner in place
Frequently asked questions
What is a local LLM on-premise?
A language model that runs on servers you control, in your own data centre or cloud account. Prompts, documents and answers are processed inside your network, not by a provider's API.
Is private AI cheaper than a hosted API?
It depends on volume. A hosted API costs per token. A local model needs hardware and operations up front and then costs about the same at any load. Steady, high volume favours local; low or uneven volume favours the API.
Can we run Qwen locally for commercial use?
The Qwen project repository states that its open-weight models are under Apache 2.0, which permits commercial use. Check the licence file of the exact model you download.
How much GPU memory does a self-hosted LLM for business need?
Enough for the weights, the context and the requests handled at the same time. Quantization reduces the memory for weights. Measure peak concurrency in a pilot before buying hardware.
Is a local LLM automatically GDPR compliant?
No. It removes the transfer to an external provider, but you still need a legal basis, access control and retention rules for the personal data you process.
Bottom line
Run a model locally when the data, a contract or the site requires it, and use a hosted API where it does not. Start with one use case, measure, and size the hardware from what you measured.
Planning a local model? Graf Clouds runs open-weight models such as Qwen, Llama and Mistral on customers' own servers, on-premises or in their own cloud account. We size and build the infrastructure, connect the model to your systems and operate it afterwards: updates, monitoring and capacity. For Presimsan, a manufacturer, we built local AI this way. See private AI and local LLM, the AI integration overview or, for hosted models, Claude and OpenAI integration. Talk to us.
Sources and further reading
- Qwen (Alibaba Cloud): project repository and licence note
- Meta: Llama community licence agreement
- Mistral AI: model overview with licences
- Google: Gemma terms of use
- Open Source Initiative: Open Source AI Definition
- vLLM: project repository
- Hugging Face: the GGUF format and quantization types
- Hugging Face: pickle scanning and model file safety