A language model running on your own server answers questions the same way a cloud service does. The difference is that the question, the attached document and the answer all stay inside the organization's network. In exchange, everything the provider used to handle moves in-house: the hardware, the choice of model, updates, access and logging.
When your own infrastructure is justified
The most common reason is the data itself. If the model has to process client data, contracts, technical documentation or source code, a cloud service means that data passes through a third party's systems. For personal data, the GDPR treats this as a processor relationship, which requires a contract and oversight of the provider. For other data, a nondisclosure agreement or an internal rule may forbid handing it over. With a locally run model, the request never leaves the network.
In an environment cut off from the internet, cloud services are not available at all, so a local model is the only option. Control matters too: with a local deployment, the organization decides which model version runs, when it is updated, what the system logs and how long it keeps the logs. A cloud model's behavior can change on the provider's schedule, which can cause trouble in a documented, approved process.
The arguments against count as well. Hardware that an organization can reasonably operate typically runs smaller models than the large cloud providers offer, so expect weaker results on tasks that need long, multi-step reasoning. For summarization, search, classification and document processing, though, these models are often good enough. Operations, security patching and troubleshooting also become the organization's job. Planning should therefore start with the actual task: what questions need answering, from what data, for how many users and with what response time.
Model size, memory and quantization
A language model's size is given as its number of parameters, often shown in billions in the model name (7B, 70B). The parameters are the model's weights. While the model runs, all of them have to be in memory, preferably the GPU's own memory, because that is what makes responses fast.
The memory needed is, to a good approximation, the number of parameters multiplied by the bytes per parameter. At 16-bit precision one parameter takes 2 bytes, so the weights of a 7-billion-parameter model take roughly 14 GB, and those of a 70-billion-parameter model around 140 GB. On top of that comes the memory used at runtime.
Context needs memory too
The model keeps the intermediate results computed for the text it has already processed (the conversation and the attached document) in a cache (the KV cache). Its size grows with the context length and with the number of requests served at the same time. A model that fits comfortably on the GPU with one user can run out of memory, or start queuing requests, when several long-document requests arrive in parallel. Sizing therefore requires knowing the expected load as well; the model size on its own is not enough.
Quantization
Quantization stores the weights at lower numerical precision, for example 8 or 4 bits instead of 16. At 8 bits the model needs about half the memory, at 4 bits about a quarter, so it runs on machines where the original version would not. The price is quality: at 8 bits the loss is typically slight, at 4 bits and below it becomes measurable and varies by task. How much this matters for a specific use can only be settled by your own testing, with sample questions taken from real work and expected answers defined in advance.
Without a GPU, the model can also run on the CPU and system memory, only much more slowly. That can be enough for a trial and for one or two users, but rarely for multi-user operation.
Open-weight models and license terms
Most models you can run locally are open-weight: the weight files can be downloaded, but that alone does not make the model open source. The training data and the training process are usually not public, and use is governed by the license that comes with the model. The Open Source AI Definition 1.0, published by the Open Source Initiative in 2024, expects open source AI to publish the code and a detailed description of the training data in addition to the weights.
Licenses vary widely. Some are standard permissive open source licenses, such as Apache 2.0 or MIT, which also allow commercial use. Others set their own terms. They may restrict commercial use, require a separate license above a certain number of users, attach an acceptable use policy, or require derived models to show their origin in the name. Some terms apply to the model's output, for example forbidding its use to train other models.
Before deployment, write down which version of which model is running, where it came from, what license it is under, and whether that license allows the intended use. A fine-tuned version of a custom-licensed model uploaded by the community is typically still bound by the original model's license terms, even if the uploader states something else. The uploader is responsible for the file's contents, and it has usually not been independently checked.
The model file is an attack surface too
A downloaded weight file is part of the software supply chain, just like a dependency or a container image. Some widely used model formats are built on Python's pickle mechanism, which can execute arbitrary code when the file is loaded. Download models only from trusted sources, verify the published checksum, and prefer a format that stores only the weights and no executable code. The glossary also has an entry on supply chain attacks.
Runtime environment and user interface
At the bottom sit the hardware and the operating system, with the GPU driver and libraries. On top of these runs the inference server, which loads the model, accepts requests, and groups requests that arrive in parallel into batches so the GPU does not sit idle. Most of these servers are reachable over HTTP, often through an interface compatible with the widely used API format of cloud language model services. That way existing tools and libraries can be connected with small changes.
Users see only the top layer of this. It can be a chat interface in the browser, a feature built into an existing application, or an internal knowledge base search. The latter typically works on the RAG (retrieval-augmented generation) principle: for each question, the system first retrieves the relevant document passages from an index and passes them to the model for the answer. The model does not have to be retrained on the organization's documents, and the answer can show which document it came from.
The runtime environment is often deployed in containers. This makes it easier to pin versions and roll back, but the container image also comes from an outside source, so it needs the same source and checksum verification as the model file.
Access control and logging
Running locally does not make a model secure. Some inference servers accept requests without authentication in their default configuration. If such a server also listens on every network interface, anyone on the internal network can use it, and in a worse case even replace the loaded model. Only the user interface or an authenticating gateway should therefore reach the API, and users should sign in through the organization's existing identity management, with SSO.
Document permissions
The biggest risk with a RAG-based search is that the system puts a document passage into an answer that the person asking could not open in its original location. The index therefore has to preserve the permissions of the source documents and run the search on behalf of the user who asked. If the index is built with a single service account that can access everything, the model can hand the entire document store to anyone who asks the right way.
Running locally does not protect against prompt injection either. If a hidden instruction makes its way into a processed document, the model may follow it on your own server too. Where the model can call tools or write to other systems, its permissions must be limited to the minimum the task needs. The article on the risks of enterprise AI adoption covers the other risks.
What to log
To investigate an incident or a complaint, you need to know who asked what, when and with which model, which document passages the model received for it, and what the answer was. With a local deployment, this log is created in the organization's own system. The log is itself sensitive, since the questions and answers contain the same data the local deployment is meant to protect. Access to it must therefore be restricted, the retention period defined, and for personal data this must also be recorded in the privacy notice.
Operational events, such as loading the model, configuration changes or failed sign-ins, belong in the central log collector together with the logs of other systems. What is worth keeping there and for how long is covered in the article on logging.
On a network cut off from the internet
On an air-gapped network, installation and updates both happen offline. The model file, container images, operating system packages and software dependencies are assembled on an internet-connected staging machine and brought into the closed network on verified media or through a one-way gateway. The checksums of the imported files must be verified again on the closed side, and the media must be scanned for malware.
A common surprise is a component that wants to download something on first start: a missing part of the model, a tokenizer or an update. On a closed network this stops with an error. The installation should therefore first be run through in a test environment that is also isolated, which will also show whether any software tries to send telemetry or whether its license requires online activation.
Updates cannot be skipped on a closed network either. Vulnerabilities in the runtime environment have to be patched like those in any other software, except that here every patch import is planned work. Write down in advance who assembles the update package, who checks it, how it is tested in the test environment, and how to roll back to the previous version if the new model gives worse answers.
Related service: Local (on-prem) AI deployment →