State why a private model
A private model may address residency, latency, customization, or control requirements. Specify the requirement and its evidence before selecting a model. Private deployment still requires secure retrieval, logging, upgrades, and evaluation; deployment location alone does not guarantee confidentiality.
Separate retrieval from generation
Benchmark retrieval quality with a fixed question set before comparing language models. If the required passage is absent, a stronger generator cannot cite it. Keep chunks, metadata, and access filters constant when testing model substitutions.
Budget hardware and throughput
Measure concurrent users, context size, output length, and required response time. Model memory and batching alter throughput. Include fallback behavior when the local service is saturated, and keep performance tests on the same prompts and documents.
Test grounded responses
Review whether each material claim is supported by a permitted retrieved passage. Track unsupported claims, missing citations, refusal quality, and language consistency. A model benchmark score alone cannot show whether the organization's own document questions are answered correctly.
Operate the full stack
Version model weights, prompts, retriever, index, and source snapshot together. Plan patching and rollback for each component. Re-evaluate after model or embedding changes because citations can move even if the user interface looks identical. Running and adapting a model inside your own environment is covered on our large language model training and fine-tuning page.
Private RAG data-path matrix
| Check | Example evidence and decision |
|---|---|
| Source | Classify the manual or contract by owner, sensitivity, region, and allowed readers. Example: a customer contract remains in a private repository; its chunks inherit the same access list before indexing, regardless of where inference runs. |
| Embedding | Record where the embedding request is processed and whether the input or vector is retained. Compare an in-region managed endpoint with self-hosted inference on the actual data path; avoid assuming that private generation also makes embedding private. |
| Retrieval | Test a permitted and a denied identity against identical query text. The search service must return only authorized passages before they enter the model context. Record index location, permission-filter mechanism, and query logs in the matrix. |
| Generation | Measure private model memory, throughput, and response time at realistic context lengths. Example decision: retain a small local model only if it meets citation and latency targets; otherwise revisit architecture, subject to the same data rules. |
| Trace | Map every saved prompt, retrieved passage, answer, and error log to a retention rule and an operator role. Test deletion of a source document through index, cache, and trace stores before claiming the system has a controlled lifecycle. |
Technical references: Microsoft Learn documentation. Related guide. Check the cited documentation against your own data, permissions and release version before applying this design.