When does running a language model on your own servers make sense?
Running a language model on your own servers makes sense when data must not leave your network, when usage is steady and high enough to keep the hardware busy, or when you need full control over which model version answers. It rarely makes sense when usage is low or irregular, when you need the strongest available models for hard tasks, or when nobody will operate the system after installation. The decision is a matter of arithmetic and risk, not principle: a hosted API bills for what you use, while your own GPUs cost the same whether they are busy or idle.
| Signal | Points to your own servers | Points to a hosted model |
|---|---|---|
| Data sensitivity | Personal, confidential or regulated data that may not leave the organisation | Public or low-sensitivity content |
| Usage pattern | Steady, predictable and high | Low, spiky or still unknown |
| Quality needed | Narrow, well-defined tasks a mid-sized open model handles well | Hard multi-step reasoning, very long documents, broad general knowledge |
| Control | The model version must stay fixed; an isolated or offline network is required | Automatic access to new models is a benefit |
| Team | People who can run GPUs, inference servers and monitoring | No operations capacity |
| Integration | Systems that exist only inside your network | Systems already in the cloud |
Many organisations end up with a mix: a model on their own servers for requests that cannot leave, and a hosted model for the rest. If you are unsure, start by measuring real usage with non-sensitive data on a hosted model. The volume you observe is a much better basis for sizing hardware than a forecast.
What exactly has to run on your servers?
A working setup is more than a model file. Plan for at least these components, because each of them needs an owner and appears somewhere in the budget:
- GPU servers with enough memory for the model and for the working memory of concurrent requests.
- An inference server that loads the model and answers requests, such as vLLM.
- An API gateway for authentication, rate limits and request logging.
- The application layer: a document search (RAG) index, prompt templates and any tools the model may call.
- Monitoring and evaluation: latency, errors and GPU usage, plus a test set that checks quality after every change.
- Storage and backups for model files, indexes and logs.
How much GPU memory does a model need?
GPU memory has to hold two things: the model's weights, and the working memory of every request being processed at the same time. Weights take roughly the number of parameters multiplied by the bytes stored per parameter. Hugging Face's Llama 3.1 announcement give concrete figures. Loading the weights alone takes about 16 GB for the 8B model at FP16, 8 GB at FP8 and 4 GB at INT4; for the 70B model, 140 GB, 70 GB and 35 GB; for the 405B model, 810 GB, 405 GB and 203 GB. The same page notes that these numbers cover only loading the checkpoint, not the space reserved for kernels and CUDA graphs.
The second part, the key-value (KV) cache, is the one buyers tend to forget. It holds the attention state of each request and grows with the length of the context. In the same source, the FP16 KV cache for a single 16,000-token context is about 1.95 GB for the 8B model and 4.88 GB for the 70B model; at 128,000 tokens it reaches 15.62 GB and 39.06 GB. The page points out that for the small model, the cache can use as much memory as the weights when the context approaches its maximum. Multiply by the number of requests you want to serve at once, and the cache often becomes the larger number.
Now compare those figures with the cards. NVIDIA lists 80 GB of memory for the H100 in its SXM form and 94 GB for the H100 NVL PCIe card; the L40S has 48 GB. A 70B model at FP16 does not fit on any one of these. At FP8 its weights fit on a single 80 GB card but leave little room for the cache, so in practice the model is spread across several GPUs. Inference servers do this with tensor parallelism, which splits one model across the cards in a server.
Lower precision, known as quantization, reduces memory but can reduce quality. Treat it as a measurable trade-off: run the quantized and unquantized versions on the same test set before you decide which one to buy hardware for.
Sizing worksheet
Fill in this worksheet with your own values before asking for hardware quotes. The third column says where each value comes from; if a line is empty, the quote you receive is based on someone else's assumption.
| Line | What to write | Where it comes from |
|---|---|---|
| 1. Use cases and users | Which tasks, how many users, working hours | Process owners |
| 2. Peak concurrent requests | Requests being answered at the same moment at peak | Usage logs or a pilot on a hosted model |
| 3. Context per request | Prompt, retrieved documents and answer, in tokens | Sample requests run through the candidate model's tokenizer |
| 4. Candidate model and precision | Parameter count; FP16, FP8 or INT4 | Model card and a quality test on your data |
| 5. Weight memory | Parameters × bytes per parameter | Model documentation |
| 6. KV cache per request | At the context length in line 3 | Model documentation or measurements from the inference server |
| 7. Total KV cache | Line 6 × line 2, as a worst case | Calculation |
| 8. Runtime headroom | Kernels, activations, fragmentation | Inference server documentation; confirm in a load test |
| 9. GPUs per replica | (Line 5 + line 7 + line 8) ÷ memory per card, rounded up to a configuration the server supports | Calculation |
| 10. Replicas | Copies needed for availability and maintenance windows | Your uptime target |
| 11. Test environment | Separate capacity to try new models and versions | Your release process |
| 12. Latency target | Time to first token and total response time | User expectations, measured in a load test |
Hypothetical example: an internal document assistant
This is a hypothetical calculation using the published Llama 3.1 figures above; real numbers depend on the model, the inference server and your traffic. Suppose an internal assistant answers questions over company documents, with a peak of 20 requests in progress at once and up to 16,000 tokens of context each (question, retrieved passages and answer), with the cache kept at FP16.
- 8B-class model at FP16: 16 GB of weights plus 20 × 1.95 GB of cache, about 39 GB, for roughly 55 GB before runtime overhead. One 80 GB or 94 GB card can hold one replica. A 48 GB card cannot, unless you quantize the weights or the cache, or lower the context limit.
- 70B-class model at FP8: 70 GB of weights plus 20 × 4.88 GB of cache, about 98 GB, for roughly 168 GB before overhead. One replica needs at least three 80 GB cards, and a second replica for availability doubles that.
This is a worst case: it assumes every request uses the full context at the same moment. vLLM's PagedAttention, described in the original research paper, allocates cache memory in blocks as requests grow, which cuts the waste caused by fragmentation and lets more requests share a card. The paper reports 2–4× higher throughput than the systems it was compared with at the same latency; your own gain depends on your traffic. The practical lesson is that choosing between an 8B and a 70B model is not only a quality decision. It changes the server count several times over, so decide it by testing both on your own tasks.
Which inference server should you use?
Choose the inference server for the hardware you have, the models you need and the interface your applications expect. vLLM is a widely used open-source option. Its documentation lists PagedAttention, continuous batching, prefix caching, an OpenAI-compatible API server, a range of quantization formats, tensor and pipeline parallelism across GPUs, and support for NVIDIA and AMD GPUs as well as other accelerators. An OpenAI-compatible interface matters to buyers: applications written for a hosted API can often be pointed at your own server with small changes, which keeps a hybrid setup manageable.
The landscape changes, so check a project's status before committing to it. Hugging Face's Text Generation Inference page now states that TGI is in maintenance mode and recommends vLLM, SGLang and local engines such as llama.cpp or MLX going forward. Engines like llama.cpp are typically used for smaller models on a single machine; load-test them before you put many concurrent users on one.
Whatever you choose, a handful of settings decide capacity. vLLM's engine arguments include the share of GPU memory the server may use, the maximum context length, the maximum number of sequences processed together, the KV cache data type and the number of GPUs a model is split across. Write these values next to the sizing worksheet; they connect the hardware quote to what users will experience.
What are the hardware and operating cost lines?
The purchase price of the GPUs is only one line of the budget. A complete comparison contains the lines below; ask every hardware and service quote to state which of them it covers and which it leaves to you.
| Cost line | Type | What drives it |
|---|---|---|
| GPUs and servers | One-time, or monthly if rented | Model size, concurrency, number of replicas |
| Storage and backup | One-time and recurring | Model files, indexes, logs and how long you keep them |
| Network and security equipment | One-time | Isolation requirements, segmentation, gateway |
| Rack space, power and cooling | Recurring | GPU power draw: NVIDIA lists up to 700 W per H100 SXM and 350 W for an L40S, before the rest of the server |
| Hardware support and spare parts | Recurring | Required response time; whether one failed card can stop the service |
| Software and support subscriptions | Recurring | Operating system, drivers, orchestration, optional commercial support for open-source tools |
| Operations staff | Recurring | Monitoring, updates, security patches, incident response |
| Evaluation and model updates | Recurring | How often you test new model versions and how large your test set is |
| Hardware refresh | Periodic | Depreciation policy and the pace of new GPU generations |
Put the hosted alternative on the same sheet: model usage by tokens, any dedicated-capacity fees, the work needed to keep personal data out of prompts, and contract and transfer review. The comparison is fair only when both columns include people, not just machines.
What does KVKK change?
Keeping the model on your own servers in Türkiye removes one specific question, the cross-border transfer at the model call, but not the rest of KVKK. In its guide on generative AI and personal data published in November 2025, the Personal Data Protection Authority states that when data controllers in Türkiye use generative AI systems through service providers based abroad, and personal data is transferred abroad as a result, the transfer must comply with Article 9 of Law No. 6698 and the related regulation. That regulation sets out transfers based on an adequacy decision, on appropriate safeguards, and incidental transfers. A model on your own servers keeps prompts and outputs off that route.
The other obligations remain. You still need a lawful basis for processing, the transparency duty under Article 10, which the same guide covers in its own section, and appropriate security measures. Prompts and answers stored in your logs are personal data whenever they contain it, so they need access control and a retention period. If any component reaches out to the internet, such as an update service, telemetry or a hosted fallback model, include that flow in the data map too. Questions about your specific data belong with your data protection counsel; the architecture's job is to make whichever route they choose provable.
Keeping data in the country and controlling the model itself are two different things. Our article on on-premise and sovereign AI explains the difference and the hybrid patterns organisations use.
What quality do you give up compared with hosted models?
You may give up some capability on hard, open-ended tasks, and you take on responsibilities the provider used to carry. Open-weight models that fit on a few GPUs can handle narrow, well-defined work well, such as classification, field extraction and answering from retrieved documents. The largest hosted models may still be stronger on long, multi-step reasoning, and quantization can lower quality further. Turkish adds its own variable: a model's general benchmark score does not tell you how it handles your documents.
| Dimension | Hosted model | Model on your servers |
|---|---|---|
| Peak capability | Access to the largest models | Limited by the hardware you buy |
| Version control | The provider may change or retire models on its own schedule | The version stays fixed until you change it |
| Safety filters | Provided and maintained by the provider | Yours to configure and test |
| New releases | Available as soon as they launch | Each upgrade is a small project: download, test, deploy |
| Data path | Leaves your network unless limited by contract and design | Stays in your network |
| Cost profile | Pay per use | Fixed capacity, used or not |
The way to settle the quality question is not a leaderboard but a test set built from your own cases and run on every candidate, hosted and local, under the same conditions. Our article on open-weight and closed models compares the two sides on control, privacy, cost and capability in more detail.
Common mistakes
- Sizing for the weights and forgetting the KV cache of concurrent users.
- Buying hardware before measuring real usage.
- Running a service people depend on from a single GPU server with no redundancy.
- Choosing a model from public benchmarks without testing it on your own data.
- Treating "on-premise" as KVKK compliance in itself.
- Leaving logs, backups and update channels out of the data-flow map.
How Hatson Tech approaches this
In our LLM training and fine-tuning work, we compare candidate open-weight models on your own Turkish test set and, where data needs to stay inside, run them on your servers or in a private cloud. The hardware follows the measured model size and load, not the other way round. If you are weighing this decision, the first meeting is free: bring an estimate of users, the documents involved and your data constraints, and we can work through the sizing worksheet together.