When is a knowledge graph better than plain RAG?
A knowledge graph helps when the answer depends on relationships spread across many documents, such as a chain from supplier to part to product, or a theme that runs through a whole collection; plain RAG is enough when the answer sits in one or a few passages that search can find. Microsoft's GraphRAG documentation names the two cases where baseline RAG falls short: it "struggles to connect the dots" when an answer requires traversing disparate pieces of information through their shared attributes, and it performs poorly when asked to understand summarised concepts across a large collection, or even across a single large document.
The research behind GraphRAG makes the second case concrete. The "From Local to Global" paper shows that conventional RAG fails on global questions about a whole corpus, such as "What are the main themes in the dataset?". Its approach builds an entity graph from the documents, pre-generates summaries for communities of closely related entities and answers a global question by combining partial answers from those summaries. On datasets in the range of one million tokens, the authors report substantial improvements over a conventional RAG baseline in the comprehensiveness and diversity of answers to such questions. That is a result for a specific question type, not a general verdict that graphs beat RAG.
"Knowledge graph" also means two different things in practice. One is a graph built from structured data you already trust, such as supplier, part and product tables in an ERP system, queried with exact rules. The other is a graph extracted from free text by a language model, which is what GraphRAG does. The first is precise but covers only what the systems hold; the second reaches into documents but inherits extraction errors. Decide which one you mean before comparing costs.
Which question shapes call for which approach?
Classify questions by shape before choosing an architecture. The table below is the decision tool: collect a few hundred real questions, label each with its shape, and the share of each shape tells you where to invest.
| Question shape | Example | Start with | When a graph earns its place |
|---|---|---|---|
| Single fact | "What does the travel policy say about approval above the limit?" | Permission-filtered hybrid search with citations | Practically never |
| Multi-part, independent | "Leave rules for the Ankara and Istanbul offices" | Query decomposition, then RAG for each part | Rarely |
| Multi-hop chain | "Which suppliers provide parts used in product family X?" | A query on structured data, if the relationships already exist in a system | When the chain crosses documents and needs entity resolution |
| Count or aggregation | "How many supplier contracts expire next year?" | Extracted fields in a table, queried exactly | A graph or a table; plain RAG cannot count reliably |
| Entity profile | "Everything we know about supplier Y" | RAG with entity filters | When an entity has many relationships across documents (GraphRAG local search) |
| Global theme | "Recurring root causes across 400 incident reports" | Map-reduce summarisation over retrieved samples, as a baseline | Corpus-level sensemaking (GraphRAG global search) |
Label by the evidence an answer needs, not by the wording. "Who supplies the valve in pump P-200?" looks like a single fact, but if the valve's part number sits in a bill of materials and the supplier in a contract, it is a two-hop chain. Count how many documents and how many hops a correct answer requires; that number, not the phrasing, decides the shape.
Two of these rows are not about graphs at all. Multi-part questions are usually solved by decomposition: Microsoft's RAG information-retrieval guide describes classifying a query as simple or complex and splitting a complex one into subqueries that are searched separately. Counting questions need structured fields, whether those fields live in a graph or in a table.
What must be decided before you extract a graph?
An extracted graph is only as useful as the rules it was built with. Write these down before the first indexing run:
- Entity types: the handful of types that matter for your questions, such as supplier, part, product, site and incident, rather than everything the model can find.
- Relationship types and direction: "supplies", "is part of", "caused", each with a direction, so a chain can be followed without guessing.
- Provenance for every edge: the source document, version and text unit, plus the extraction run and prompt version. An edge without a source passage cannot be checked or cited.
- Entity resolution rules: which identifiers decide that two mentions are the same entity, such as a tax number or a part number, and what happens when none is available. When identity is uncertain, keep two candidates rather than merging them.
- Refresh and deletion rules: what happens to entities, edges and summaries when a document is updated, renamed or withdrawn, and how quickly.
- Conflict handling: when two documents state contradictory relationships, keep both with their dates and sources instead of letting the latest extraction silently win.
The same principle applies at query time. When an answer relies on an extracted relationship, show the passages that support it next to the answer; if the extraction is uncertain, presenting the underlying text is more useful than asserting the relationship as fact.
What does a knowledge graph really cost?
An LLM-extracted graph moves a large part of the work to indexing time. GraphRAG's pipeline extracts entities, relationships and key claims from every text unit, clusters the graph hierarchically with the Leiden technique and generates a summary for each community from the bottom up. The project's README carries an explicit warning that indexing can be an expensive operation, advises reading the documentation on the process and its costs and starting small, and strongly recommends tuning the extraction prompts to your own data.
The trade-off between paying at index time and paying at query time shows in Microsoft Research's own follow-up work. Its LazyGraphRAG post (November 2024) describes an approach that defers language-model use to query time, with indexing costs identical to vector RAG and 0.1% of the cost of full GraphRAG. Whichever variant you consider, put four costs side by side:
- Build: model calls for extraction and summaries across the whole collection.
- Refresh: when documents change, entities, edges and community summaries must change too. A weekly policy update can touch summaries far from the edited page.
- Quality control: extraction errors become "facts" in the graph. Entity resolution, such as one supplier under three spellings or one abbreviation used for two systems, needs rules and review.
- Permissions: a community summary may blend documents with different access rules. If a summary draws on a restricted contract, anyone who can read the summary can read part of that contract. Build summaries per permission scope, or keep restricted documents out of summarisation.
How do you test whether a graph is worth it?
- Build the baseline first: permission-filtered hybrid search with reranking, plus decomposition for multi-part questions.
- Label real questions by shape and record the share of each. If nine in ten are single facts, a graph will barely change the user's experience.
- Pilot the graph on one slice, such as one document family, and only for the shapes where the baseline fails.
- Measure the same things on both paths: supported-answer rate, citation correctness, permission correctness, latency, build cost and refresh cost. For global questions, ask reviewers to compare pairs of answers for comprehensiveness and to check every reported theme against its sources.
- Route by shape. Send each question shape to the path that won for it, log the route, and fall back to plain retrieval when the graph path fails.
Hypothetical example: incident reports and suppliers
A hypothetical manufacturer has 400 incident reports, a few hundred supplier contracts and a parts list in its ERP system. The team labels 120 real questions: 84 single facts, 12 multi-part questions, 14 supplier chains and 10 global themes. The results below are illustrative:
| Shape | Baseline result | Graph pilot | Decision |
|---|---|---|---|
| Single fact (84) | Supported in most cases | Not tested | Stay on hybrid RAG |
| Multi-part (12) | Improved by decomposition | Not tested | Decomposition |
| Supplier chain (14) | Often misses one link in the chain | Not needed: the ERP already holds the supplier–part links | A structured query tool over ERP tables, with contract passages as citations |
| Global theme (10) | Narrow themes drawn from the top 50 passages | Broader themes; two of them unsupported on review | GraphRAG pilot on incident reports only, with every theme shown next to its source reports |
The outcome is a router, not a platform change. Most questions stay on the cheaper retrieval path, chain questions go to data that is already structured, and a graph is built only where it changed the answer: the incident reports.
Common mistakes
- Building a graph before classifying real questions by shape.
- Extracting from text relationships that already exist, cleanly, in a business system.
- Treating extracted edges as facts without links to their source passages.
- Generating community summaries across documents with different permissions.
- Comparing a tuned graph pipeline against an untuned RAG baseline.
Where should you start?
Start by labelling one hundred real questions by shape; the share of chain and global questions decides whether a graph is worth piloting at all. To measure both paths on the same terms, see our article on RAG evaluation metrics. The comparison table on our RAG and semantic search service page lists GraphRAG as the better fit for questions about relationships between people, organisations and events.