A Different Question From RAG Versus Fine-Tuning
There are two separate architectural questions that get confused constantly, and answering the wrong one wastes months. The first is where knowledge should live: baked into model weights through training, or held outside the model in an index. That is the training-versus-retrieval question, and we have covered it separately in RAG versus fine-tuning. It is still a live question and its answer has not changed much. The second question is newer, and it is the one this article is about: given that the knowledge lives outside the model, should it arrive in the prompt or through a retrieval step?
Three years ago this second question did not exist in any practical sense, because context windows were small enough that the answer was forced. You could not paste a corpus into a prompt, so you built retrieval, and the only debate was about chunk sizes. Windows have since grown by two to three orders of magnitude and long-context recall has improved substantially alongside them, which means that for the first time you can genuinely choose. A great many documents that used to require a pipeline now fit in a single request with room to spare.
The conclusion a lot of people drew from that was that retrieval is obsolete. It is not, and the reasons have almost nothing to do with model capability. They are about cost that scales with every request rather than once, about access control that a prompt cannot express, about citations that an auditor will ask for, and about corpora that are larger than any window will ever be. What follows is the comparison as an engineer should make it: what changed, what did not, where each approach genuinely wins, and the hybrid that most production systems end up at anyway.
What Actually Changed, and What Did Not
Two things genuinely changed. Context windows grew by orders of magnitude, which moved a large class of documents from impossible to trivial. And long-context recall improved: models today make substantially better use of information deep inside a long input than earlier generations did, which is why the naive test of pasting a big document and asking a question now usually works when it used to fail. If your mental model of long context is from an older generation of models, it is out of date, and you should re-run your own test before dismissing the approach.
Two things did not change at all, and they are both economics rather than capability. First, cost scales with what you send. Every token in the prompt is a token you pay for, on every single request, forever. A retrieval index is paid for once at ingestion and then queried cheaply; a long-context prompt is paid for again each time somebody asks anything. Second, latency scales with input length too. Prefill is real work, and time-to-first-token grows with the size of the input, which is why the same question against a small prompt feels instant and against a very large one does not.
So the honest summary is that long context removed a capability constraint and left the operational constraints exactly where they were. That is a genuinely important change, because it means retrieval no longer has to be justified on the grounds that the model cannot handle the text. It has to be justified on cost, permissions, freshness and auditability instead, which is a much more specific conversation and a much easier one to have with a finance or compliance stakeholder. Deciding by architecture fashion rather than by these four variables is how teams end up with a pipeline they did not need or a bill they did not model.
Context Rot and Position Effects
There is a well-documented behaviour that practitioners call context rot, and that we can render in Turkish as baglam curumesi: as the amount of context grows, the model's effective use of it degrades, even when the window nominally supports the length. This is not the same as the model failing to find a fact. It is subtler and more damaging in production: answers get vaguer, hedge more, drop qualifications, and blend details from different documents. The system does not report an error. It reports a slightly worse answer, which is exactly the failure mode that survives casual testing and shows up in user complaints months later.
Position effects compound this. Information placed at the beginning and the end of a long input is attended to more reliably than information sitting in the middle. In practice this means that where you put something matters as much as whether you included it, which is an uncomfortable property for anyone who thought a big window meant they could stop thinking about prompt structure. The key clause of a contract buried in the middle of a two-hundred-page dump is materially less likely to drive the answer than the same clause retrieved and placed at the top of a short prompt.
Then there are distractors. Irrelevant content that is topically similar to the question does not sit harmlessly in the background; it actively pulls the answer around. More context is not monotonically better, and this is the single most counter-intuitive fact in the whole comparison. The operational signal is easy to recognise once you know to look for it: answers get vaguer as you add documents rather than sharper. Curation beats dumping even inside a window that fits, which is why disciplined context engineering did not become less important when windows grew. It became the thing that separates a system that works from one that merely runs.
The Honest Cost Comparison
Set the two approaches side by side without loyalty to either. Retrieval has an up-front cost: embedding the corpus, storing the vectors, running the infrastructure, plus incremental cost as documents change. It then has a small marginal cost per query, typically an embedding call and a search that together run in the 20 to 150 millisecond range and cost a rounding error in tokens. Long context has no up-front cost at all, which is genuinely attractive, and a marginal cost per query equal to sending the entire corpus, every time, for every user.
The crossover arithmetic is simple. If indexing a corpus costs some one-time amount and sending it costs some amount per request, the two approaches break even after a number of requests equal to the first divided by the second. Because indexing is a one-time cost spread across all future queries while sending is paid per query per user, that break-even number is small for essentially any corpus of consequence. In the systems we have modelled, it lands in the tens to low hundreds of requests. A workload that will ever be used by more than a handful of people crosses it in the first week.
The multiplier that people miss is concurrency. Ten users asking the same question of a long-context prompt pay ten times for the same tokens. Ten users querying an index pay for ten cheap searches against work that was done once. Multiply that across a support tool used by a thousand people a day and the difference stops being an optimisation and becomes the entire economics of the product. If you are building a per-workflow cost model, this is exactly the kind of decision that shows up in model routing and cost control, and it should be modelled before the architecture is chosen rather than discovered afterwards.
When Long Context Genuinely Wins
The clearest win is a single bounded document. One contract, one regulatory filing, one meeting transcript, one technical specification, one module of a codebase. If the whole thing fits and the question is about that thing, building a retrieval pipeline is architectural theatre. You will spend three weeks on chunking, embedding and evaluation to produce something worse than pasting the document in, because retrieval necessarily fragments a document that the model could have read whole. Our standing advice is blunt: do not build RAG to answer questions about one PDF.
The second win is one-off analysis with no reuse. Due diligence on a single acquisition target, a legal opinion on one case file, an audit of one year's records. The defining feature is that the corpus will be queried a handful of times and then never again, which puts you on the wrong side of the break-even calculation. The third is prototyping, and it is the one we recommend most often. Before you build any pipeline, prove the task is possible by doing it in long context on a sample. If the model cannot do the task with the documents handed to it directly, retrieval will not rescue it.
The fourth win is a category rather than a scale: questions that require global structure. Summarise this entire document. Find every inconsistency across these clauses. Trace how this argument develops. Identify what is missing relative to a template. Retrieval fragments these questions by construction, because there is no query whose top results contain the whole. A model reading the complete text has an advantage no retrieval configuration can reproduce, and trying to reproduce it with an ever-larger top-k is a well-trodden path to a slow, expensive system that still gives partial answers.
When Retrieval Wins
The first and most common reason is simple size. Enterprise corpora are not context-window problems. The pilots we run typically start at 50,000 to 200,000 documents, and the production systems that follow are larger, growing, and never going to fit in any window that will exist. Once the corpus exceeds the window, the question stops being architectural preference and becomes arithmetic. Everything after this point in the list is a reason retrieval wins even when the corpus would fit, which is the more interesting case. Everything after this point is a reason retrieval wins even when the corpus would fit, which is the more interesting case and the one worth reading carefully.
Access control is the strongest of those reasons. A prompt has no permission model. If ten users have ten different entitlements over the same corpus, you cannot express that in a prompt; you would need a different prompt per entitlement set, which is retrieval with extra steps and worse ergonomics. Retrieval carries document-level permissions into a query-time filter, which is the only design that survives a security review. Freshness is the second: an index updates incrementally when a document changes, while a long-context prompt is rebuilt in full, and in a corpus with daily changes that difference is the whole operational cost.
Then citations and auditability. Retrieval gives you provenance almost for free: this answer came from this chunk, of this document, in this version, last modified on this date. Long context gives you an answer that mentions a source, which is not the same thing and cannot be verified programmatically. In regulated Turkish deployments this matters concretely, because Article 12 of Law No. 6698 puts a data-security obligation on the veri sorumlusu and any serious audit asks which record fed which decision. The vector database is where that answer lives, and cost at volume, discussed above, is simply the fourth reason on top.
The Hybrid Almost Everyone Converges On
Production systems built after the window expansion do not choose. They retrieve, and then they are generous with what they send. The old discipline of returning three chunks of three hundred tokens existed because context was scarce and expensive; that scarcity is gone, and the configurations that work now look completely different. Larger windows did not make retrieval obsolete. They made retrieval better, by removing the constraint that forced it to be stingy at precisely the moment when precision mattered most. The configurations that work now look completely different from the ones documented three years ago, and copying an older reference architecture is a reliable way to build something needlessly stingy.
A configuration we commonly run looks like this. Hybrid search, combining lexical and vector matching, returns a candidate set of roughly the top fifty. A reranker cuts that to the ten to twenty that genuinely answer the question, which adds roughly 100 to 400 milliseconds depending on model and batch size. Each surviving chunk is then expanded to its parent section rather than sent as a fragment, so the model receives coherent passages of a few thousand tokens rather than orphaned paragraphs. The details of that pipeline are covered in hybrid search and reranking, and the expansion strategy depends heavily on how you did document chunking at scale.
The design principle underneath is worth stating plainly: use retrieval for precision and context for robustness. Retrieval decides what is relevant, which is a job it does far better than the model's attention does across a hundred thousand tokens of noise. Context length then buys you tolerance for imperfect retrieval, because sending twenty candidates instead of three means a mediocre ranking still contains the answer somewhere the model will find. The two mechanisms cover each other's failure modes, which is why this shape keeps being reinvented independently by teams who started from opposite positions.
Caching Changes the Arithmetic
Prompt caching is the one development that genuinely shifts the comparison toward long context, and it does so in a narrow and specific case. Providers cache a stable prefix and charge substantially less for cached input tokens on subsequent requests that share it. If the same large document is queried repeatedly within the cache lifetime, you pay full price once and a fraction thereafter, which changes a per-request cost into something closer to a per-session cost. That is a real and material difference, and any comparison written before caching existed is now wrong.
The limits matter as much as the effect. Cache lifetimes are short, typically measured in minutes rather than hours, so caching helps enormously within a single working session and does nothing across a user base spread over a day. A lawyer working through one large contract in a thirty-minute session is close to the ideal case for cached long context. A support assistant answering ten thousand different users about the same knowledge base is the opposite: every request is a cold start, and you pay the full corpus every time.
There is also a constraint that quietly disqualifies caching for most enterprise deployments, and it is worth understanding before you build a plan around it. The cached prefix must be identical token for token. A corpus prompt filtered by each user's entitlements is not identical between users, by definition, which means the moment you have per-document access control you have lost the shared cache. You can cache per user, but then you are caching a corpus per person and the economics collapse. Multi-tenant systems face the same wall for the same reason, and it is not a limitation any amount of prompt engineering removes.
Graph Retrieval, Structured Queries and a Decision Tree
There is a class of question that neither vector search nor long context handles well, and it is worth naming because teams lose months to it. Vector search answers what is semantically similar to this question. It cannot answer how many, which ones changed between two versions, what is the chain of ownership, or list every contract containing this clause type. Those are aggregation, traversal and filtering problems, and similarity is the wrong tool. Long context does not rescue them either: a model reading five hundred contracts to count something will miscount, confidently, and you will not know which way.
The answers are structural. Metadata filters and ordinary SQL over fields extracted at ingestion handle counting, filtering and date ranges properly, and should be the first thing you reach for. Graph retrieval over entities and relations, in the style popularised by GraphRAG, handles traversal questions such as ownership chains, citation networks and dependency paths. The unglamorous version of this advice is that a good retrieval system usually contains a small amount of boring structured data engineering, and skipping it is why so many pilots answer well-formed questions beautifully and fall apart on the first question that involves a number.
So here is the decision tree in prose. Start with corpus size: if it does not fit in a window, you are retrieving, and the rest of the tree is about how. If it does fit, ask about query volume: one-off or a handful of queries favours long context, repeated querying by many users favours an index, and the crossover is low. Then ask about access control: any per-document permission requirement forces retrieval, because a prompt cannot express one. Then audit: if anyone will ever need to know which source produced which claim, retrieval gives you that structurally and long context does not.
Two more branches finish it. Ask what kind of question dominates. Local fact lookup is retrieval's home ground. Global structure, whole-document summarisation and consistency checking favour long context. Aggregation and traversal need structured or graph retrieval regardless of which of the two you chose. Finally ask about freshness: if the corpus changes daily, an incrementally updated index is operationally cheaper than rebuilding a prompt. Most real systems answer differently on different branches, which is precisely why the hybrid exists and why the honest answer to the title question is almost always both.
The Verification Gap, and How We Decide
Neither approach verifies anything. This is the part that gets lost in architecture debates, and it is the part that determines whether a system is safe to put in front of a customer. A long-context model can read a document correctly and still produce a claim the document does not support. A retrieval system can ground on the wrong chunk and cite it, which is arguably worse, because a citation looks like evidence and invites less scrutiny than an unsourced assertion. The failure rate differs between the two approaches; the existence of the failure does not, and it is the same hallucination problem in both cases.
What closes the gap is measurement, not architecture. A fixed evaluation set with known answers, run on every change. A groundedness check that asks whether each claim in the output is actually supported by the retrieved span, rather than merely adjacent to it. Citation validation that verifies the cited passage contains the assertion. Human review on anything with legal or financial consequence. These are covered in more depth in RAG evaluation metrics, and they apply identically whether the text arrived by retrieval or by being pasted into the prompt. None of these are architecture decisions, and all of them are cheaper to build than a migration between architectures.
Our own sequence is deliberately unromantic. We prototype in long context first, because it takes an afternoon and answers the only question that matters at that stage, which is whether the task is possible at all. We measure quality, cost per query and latency on a realistic sample. Then we build retrieval when the numbers say so, which in practice means when the corpus does not fit, when per-document permissions are required, when citations are needed, or when query volume crosses the break-even. A first pilot on a 50,000 to 200,000 document corpus typically takes 6 to 10 weeks to reach a measured baseline.
The split is visible in our own products. In caseon.ai and DiligenceAI, reviewing a single contract end to end is a long-context job, because the questions are about the whole document and fragmenting it loses exactly the cross-references that matter. Searching across a body of case law or a portfolio of hundreds of agreements is a retrieval job, because it does not fit, because access differs by user, and because every answer has to cite its source. The same product does both, routed by the shape of the question. If you are scoping something similar, our RAG and semantic search work usually begins with exactly this measurement rather than with an architecture decision made in advance.