Running a Language Model on Your Own Servers: Hardware and Cost Lines
Business

Running a Language Model on Your Own Servers: Hardware and Cost Lines

GPU memory is only one line of the budget. A sizing worksheet, the cost lines a complete quote should cover, and what KVKK does and does not change.

Short answer

Running a language model on your own servers makes sense when data must stay in your network, usage is steady and high, or you need a fixed model version. Size the GPUs for both the model weights and the working memory (KV cache) of concurrent requests, budget for power, support, staff and model updates as well as hardware, and test candidate models on your own data before buying.

When does running a language model on your own servers make sense?

Running a language model on your own servers makes sense when data must not leave your network, when usage is steady and high enough to keep the hardware busy, or when you need full control over which model version answers. It rarely makes sense when usage is low or irregular, when you need the strongest available models for hard tasks, or when nobody will operate the system after installation. The decision is a matter of arithmetic and risk, not principle: a hosted API bills for what you use, while your own GPUs cost the same whether they are busy or idle.

SignalPoints to your own serversPoints to a hosted model
Data sensitivityPersonal, confidential or regulated data that may not leave the organisationPublic or low-sensitivity content
Usage patternSteady, predictable and highLow, spiky or still unknown
Quality neededNarrow, well-defined tasks a mid-sized open model handles wellHard multi-step reasoning, very long documents, broad general knowledge
ControlThe model version must stay fixed; an isolated or offline network is requiredAutomatic access to new models is a benefit
TeamPeople who can run GPUs, inference servers and monitoringNo operations capacity
IntegrationSystems that exist only inside your networkSystems already in the cloud

Many organisations end up with a mix: a model on their own servers for requests that cannot leave, and a hosted model for the rest. If you are unsure, start by measuring real usage with non-sensitive data on a hosted model. The volume you observe is a much better basis for sizing hardware than a forecast.

What exactly has to run on your servers?

A working setup is more than a model file. Plan for at least these components, because each of them needs an owner and appears somewhere in the budget:

  • GPU servers with enough memory for the model and for the working memory of concurrent requests.
  • An inference server that loads the model and answers requests, such as vLLM.
  • An API gateway for authentication, rate limits and request logging.
  • The application layer: a document search (RAG) index, prompt templates and any tools the model may call.
  • Monitoring and evaluation: latency, errors and GPU usage, plus a test set that checks quality after every change.
  • Storage and backups for model files, indexes and logs.

How much GPU memory does a model need?

GPU memory has to hold two things: the model's weights, and the working memory of every request being processed at the same time. Weights take roughly the number of parameters multiplied by the bytes stored per parameter. Hugging Face's Llama 3.1 announcement give concrete figures. Loading the weights alone takes about 16 GB for the 8B model at FP16, 8 GB at FP8 and 4 GB at INT4; for the 70B model, 140 GB, 70 GB and 35 GB; for the 405B model, 810 GB, 405 GB and 203 GB. The same page notes that these numbers cover only loading the checkpoint, not the space reserved for kernels and CUDA graphs.

The second part, the key-value (KV) cache, is the one buyers tend to forget. It holds the attention state of each request and grows with the length of the context. In the same source, the FP16 KV cache for a single 16,000-token context is about 1.95 GB for the 8B model and 4.88 GB for the 70B model; at 128,000 tokens it reaches 15.62 GB and 39.06 GB. The page points out that for the small model, the cache can use as much memory as the weights when the context approaches its maximum. Multiply by the number of requests you want to serve at once, and the cache often becomes the larger number.

Now compare those figures with the cards. NVIDIA lists 80 GB of memory for the H100 in its SXM form and 94 GB for the H100 NVL PCIe card; the L40S has 48 GB. A 70B model at FP16 does not fit on any one of these. At FP8 its weights fit on a single 80 GB card but leave little room for the cache, so in practice the model is spread across several GPUs. Inference servers do this with tensor parallelism, which splits one model across the cards in a server.

Lower precision, known as quantization, reduces memory but can reduce quality. Treat it as a measurable trade-off: run the quantized and unquantized versions on the same test set before you decide which one to buy hardware for.

Sizing worksheet

Fill in this worksheet with your own values before asking for hardware quotes. The third column says where each value comes from; if a line is empty, the quote you receive is based on someone else's assumption.

LineWhat to writeWhere it comes from
1. Use cases and usersWhich tasks, how many users, working hoursProcess owners
2. Peak concurrent requestsRequests being answered at the same moment at peakUsage logs or a pilot on a hosted model
3. Context per requestPrompt, retrieved documents and answer, in tokensSample requests run through the candidate model's tokenizer
4. Candidate model and precisionParameter count; FP16, FP8 or INT4Model card and a quality test on your data
5. Weight memoryParameters × bytes per parameterModel documentation
6. KV cache per requestAt the context length in line 3Model documentation or measurements from the inference server
7. Total KV cacheLine 6 × line 2, as a worst caseCalculation
8. Runtime headroomKernels, activations, fragmentationInference server documentation; confirm in a load test
9. GPUs per replica(Line 5 + line 7 + line 8) ÷ memory per card, rounded up to a configuration the server supportsCalculation
10. ReplicasCopies needed for availability and maintenance windowsYour uptime target
11. Test environmentSeparate capacity to try new models and versionsYour release process
12. Latency targetTime to first token and total response timeUser expectations, measured in a load test

Hypothetical example: an internal document assistant

This is a hypothetical calculation using the published Llama 3.1 figures above; real numbers depend on the model, the inference server and your traffic. Suppose an internal assistant answers questions over company documents, with a peak of 20 requests in progress at once and up to 16,000 tokens of context each (question, retrieved passages and answer), with the cache kept at FP16.

  • 8B-class model at FP16: 16 GB of weights plus 20 × 1.95 GB of cache, about 39 GB, for roughly 55 GB before runtime overhead. One 80 GB or 94 GB card can hold one replica. A 48 GB card cannot, unless you quantize the weights or the cache, or lower the context limit.
  • 70B-class model at FP8: 70 GB of weights plus 20 × 4.88 GB of cache, about 98 GB, for roughly 168 GB before overhead. One replica needs at least three 80 GB cards, and a second replica for availability doubles that.

This is a worst case: it assumes every request uses the full context at the same moment. vLLM's PagedAttention, described in the original research paper, allocates cache memory in blocks as requests grow, which cuts the waste caused by fragmentation and lets more requests share a card. The paper reports 2–4× higher throughput than the systems it was compared with at the same latency; your own gain depends on your traffic. The practical lesson is that choosing between an 8B and a 70B model is not only a quality decision. It changes the server count several times over, so decide it by testing both on your own tasks.

Which inference server should you use?

Choose the inference server for the hardware you have, the models you need and the interface your applications expect. vLLM is a widely used open-source option. Its documentation lists PagedAttention, continuous batching, prefix caching, an OpenAI-compatible API server, a range of quantization formats, tensor and pipeline parallelism across GPUs, and support for NVIDIA and AMD GPUs as well as other accelerators. An OpenAI-compatible interface matters to buyers: applications written for a hosted API can often be pointed at your own server with small changes, which keeps a hybrid setup manageable.

The landscape changes, so check a project's status before committing to it. Hugging Face's Text Generation Inference page now states that TGI is in maintenance mode and recommends vLLM, SGLang and local engines such as llama.cpp or MLX going forward. Engines like llama.cpp are typically used for smaller models on a single machine; load-test them before you put many concurrent users on one.

Whatever you choose, a handful of settings decide capacity. vLLM's engine arguments include the share of GPU memory the server may use, the maximum context length, the maximum number of sequences processed together, the KV cache data type and the number of GPUs a model is split across. Write these values next to the sizing worksheet; they connect the hardware quote to what users will experience.

What are the hardware and operating cost lines?

The purchase price of the GPUs is only one line of the budget. A complete comparison contains the lines below; ask every hardware and service quote to state which of them it covers and which it leaves to you.

Cost lineTypeWhat drives it
GPUs and serversOne-time, or monthly if rentedModel size, concurrency, number of replicas
Storage and backupOne-time and recurringModel files, indexes, logs and how long you keep them
Network and security equipmentOne-timeIsolation requirements, segmentation, gateway
Rack space, power and coolingRecurringGPU power draw: NVIDIA lists up to 700 W per H100 SXM and 350 W for an L40S, before the rest of the server
Hardware support and spare partsRecurringRequired response time; whether one failed card can stop the service
Software and support subscriptionsRecurringOperating system, drivers, orchestration, optional commercial support for open-source tools
Operations staffRecurringMonitoring, updates, security patches, incident response
Evaluation and model updatesRecurringHow often you test new model versions and how large your test set is
Hardware refreshPeriodicDepreciation policy and the pace of new GPU generations

Put the hosted alternative on the same sheet: model usage by tokens, any dedicated-capacity fees, the work needed to keep personal data out of prompts, and contract and transfer review. The comparison is fair only when both columns include people, not just machines.

What does KVKK change?

Keeping the model on your own servers in Türkiye removes one specific question, the cross-border transfer at the model call, but not the rest of KVKK. In its guide on generative AI and personal data published in November 2025, the Personal Data Protection Authority states that when data controllers in Türkiye use generative AI systems through service providers based abroad, and personal data is transferred abroad as a result, the transfer must comply with Article 9 of Law No. 6698 and the related regulation. That regulation sets out transfers based on an adequacy decision, on appropriate safeguards, and incidental transfers. A model on your own servers keeps prompts and outputs off that route.

The other obligations remain. You still need a lawful basis for processing, the transparency duty under Article 10, which the same guide covers in its own section, and appropriate security measures. Prompts and answers stored in your logs are personal data whenever they contain it, so they need access control and a retention period. If any component reaches out to the internet, such as an update service, telemetry or a hosted fallback model, include that flow in the data map too. Questions about your specific data belong with your data protection counsel; the architecture's job is to make whichever route they choose provable.

Keeping data in the country and controlling the model itself are two different things. Our article on on-premise and sovereign AI explains the difference and the hybrid patterns organisations use.

What quality do you give up compared with hosted models?

You may give up some capability on hard, open-ended tasks, and you take on responsibilities the provider used to carry. Open-weight models that fit on a few GPUs can handle narrow, well-defined work well, such as classification, field extraction and answering from retrieved documents. The largest hosted models may still be stronger on long, multi-step reasoning, and quantization can lower quality further. Turkish adds its own variable: a model's general benchmark score does not tell you how it handles your documents.

DimensionHosted modelModel on your servers
Peak capabilityAccess to the largest modelsLimited by the hardware you buy
Version controlThe provider may change or retire models on its own scheduleThe version stays fixed until you change it
Safety filtersProvided and maintained by the providerYours to configure and test
New releasesAvailable as soon as they launchEach upgrade is a small project: download, test, deploy
Data pathLeaves your network unless limited by contract and designStays in your network
Cost profilePay per useFixed capacity, used or not

The way to settle the quality question is not a leaderboard but a test set built from your own cases and run on every candidate, hosted and local, under the same conditions. Our article on open-weight and closed models compares the two sides on control, privacy, cost and capability in more detail.

Common mistakes

  • Sizing for the weights and forgetting the KV cache of concurrent users.
  • Buying hardware before measuring real usage.
  • Running a service people depend on from a single GPU server with no redundancy.
  • Choosing a model from public benchmarks without testing it on your own data.
  • Treating "on-premise" as KVKK compliance in itself.
  • Leaving logs, backups and update channels out of the data-flow map.

How Hatson Tech approaches this

In our LLM training and fine-tuning work, we compare candidate open-weight models on your own Turkish test set and, where data needs to stay inside, run them on your servers or in a private cloud. The hardware follows the measured model size and load, not the other way round. If you are weighing this decision, the first meeting is free: bring an estimate of users, the documents involved and your data constraints, and we can work through the sizing worksheet together.

Key takeaways

  • Self-host when data must stay inside, usage is steady and high, or the model version must stay fixed; otherwise start on a hosted model and measure.
  • GPU memory must hold the weights plus the KV cache of every concurrent request, and the cache often dominates.
  • Fill in a sizing worksheet from measured usage before asking for hardware quotes.
  • Budget for power, cooling, support, staff, evaluation and hardware refresh, not only GPUs.
  • Running on your own servers removes the cross-border transfer at the model call, not the rest of KVKK.
  • Settle the quality question with a test set from your own cases, run on hosted and local candidates alike.

Frequently asked questions

How much GPU memory does a 70B model need?

For the weights alone, Hugging Face's Llama 3.1 figures are about 140 GB at FP16, 70 GB at FP8 and 35 GB at INT4. The KV cache of concurrent requests and runtime overhead come on top, so a 70B model usually runs across several GPUs.

Can we run a language model without GPUs?

Small models can run on CPUs with engines such as llama.cpp, which suits light use on a single machine. Larger models and many concurrent users normally need GPUs; load-test any setup with realistic traffic before committing to it.

Is vLLM free to use?

vLLM is an open-source project, so there is no licence fee for the software itself. You still pay for hardware, operations and any commercial support you choose, and you need to check the licence of each model you serve separately.

Does running the model on our own servers make us KVKK compliant?

No. It keeps prompts off the cross-border transfer route at the model call, but you still need a lawful basis, the transparency notice, security measures and retention rules for logs. Confirm the specifics with your data protection counsel.

Will an open-weight model be as good as a hosted one?

For narrow, well-defined tasks it can be good enough, while the largest hosted models may still be stronger at long, multi-step reasoning. The only reliable answer comes from testing both on the same set of your own cases.

Should we buy or rent GPUs?

Renting turns a large up-front cost into a monthly one and leaves a way out while usage is uncertain. Buying makes more sense once usage is measured, steady and high enough to keep the hardware busy. Compare both using the full list of cost lines, including staff.

All articles