Pure Vector Similarity Has Predictable Blind Spots
Dense retrieval is genuinely good at the thing it was built for: finding text that means the same as the query while using different words. But an embedding is a lossy compression of a passage into a few hundred or a few thousand dimensions, and compression discards whatever the training objective did not reward. What it discards, reliably, is precision on rare tokens. A part number, a case reference, a tax code, a customer identifier and an unusual surname all collapse into roughly the same neighbourhood as their near-neighbours, because the model never had a reason to keep them apart.
The failure is easy to reproduce. Search a technical corpus for a specific part number and a dense retriever will happily return the same component in a different revision, a similar part from another product line, while the exact match sits at rank fourteen. Search a legal corpus for a specific article number and you get the neighbouring articles, because they are lexically and semantically adjacent. The user is not asking for something similar. They are asking for exactly that string, and the retriever has no mechanism to know that exactness is what matters here.
Negation and quantity are the second family of failures, and they are worse because they are silent. The embeddings of a sentence saying a treatment is recommended and one saying it is not recommended are extremely close. The same holds for thresholds: text about limits above 500,000 lira and text about limits below 500,000 lira are near neighbours. A dense retriever will return both, and if only one enters the context window it may well be the wrong one. This is a retrieval bug that presents as a generation bug, and teams routinely spend weeks rewriting prompts to fix it.
None of this means dense retrieval is wrong. It means it is one signal among several, and the correct architecture combines it with a signal that is exact where it is fuzzy. If you have not yet built a first vector-based system, the groundwork is covered in vector databases and semantic search and in how tokens, embeddings and vectors relate. This article assumes that system exists, works acceptably, and has hit the ceiling that pure similarity always eventually imposes. That combination is what the rest of this article describes and how to measure it.
What Lexical Retrieval Is Actually Good At
BM25 is a term-frequency ranking function that has been the default in production search for decades. It scores a document by how often the query terms appear in it, damped so that repetition has diminishing returns, weighted by how rare each term is across the corpus, and normalised for document length. That last property, rare terms score high, is exactly the behaviour you want for identifiers. The token that appears in three documents out of two million carries enormous weight, which is precisely the opposite of what an embedding does with it.
Practically, this means BM25 wins on exact identifiers, quoted phrases, rare proper nouns, code symbols, product SKUs, statute and clause references, and any query where the user typed a string they expect to see reproduced verbatim. It also wins whenever the corpus contains vocabulary the embedding model never saw during training, which for a Turkish enterprise corpus full of internal project codenames and domain jargon is a substantial fraction of the interesting queries. And it costs almost nothing: an inverted index in Elasticsearch, OpenSearch or a Postgres full-text column adds a few milliseconds per query and negligible memory compared with the vector index.
It loses on everything dense retrieval was invented for. A user who asks about staff travel reimbursement will not match a document titled expense policy for business trips, because they share no terms. This complementarity is the entire argument for hybrid retrieval: the two methods fail on disjoint sets of queries. In our measurements on mixed enterprise corpora, moving from dense-only to a properly fused hybrid typically lifts recall@10 by five to fifteen points, and the gain is concentrated almost entirely in the identifier and rare-term queries where dense-only was returning nothing useful at all.
One configuration warning specific to Turkish. BM25 depends on tokenisation and normalisation, so an index configured with a default English analyser over Turkish text will treat each inflected surface form as an unrelated term and score them independently. You need a Turkish analyser with stemming, correct dotted and dotless i handling, and a locale-aware lowercase filter. This is a five-line configuration change that we have seen account for double-digit recall differences on Turkish corpora, and it is one of the first things worth checking on any existing Turkish search deployment.
Fusion: Reciprocal Rank Fusion versus Weighted Scores
Once you run two retrievers you have to merge two result lists, and the merge strategy matters more than most teams expect. The problem is that BM25 scores and cosine similarities are not comparable. BM25 is unbounded and corpus-dependent; cosine similarity sits in a narrow band, often between 0.6 and 0.9 for anything plausible. Adding them directly is meaningless. Multiplying them is worse. You need either a principled normalisation or a method that ignores scores entirely. Getting this wrong is one of the quieter ways a hybrid system underperforms a pure dense one.
Reciprocal rank fusion ignores scores and uses only positions. Each document gets a contribution of one divided by a constant plus its rank in each list, with the constant conventionally set to 60, and the contributions are summed. It is the right default for two reasons: it needs no tuning, and it is completely robust to the two retrievers having incompatible score distributions. Its weakness is that it throws away magnitude, so a document that is an overwhelming lexical match and one that is a marginal match both contribute the same amount if they occupy the same rank.
Weighted score blending keeps magnitude but requires normalisation first. Min-max normalise each result list to a zero-to-one range within the query, or use a z-score against the score distribution of that query's candidates, then combine with a weight. A starting point of 0.6 to 0.8 on the dense side and the remainder on lexical is reasonable for general document corpora, shifting toward lexical for corpora dominated by identifiers and technical references. Normalise per query, never globally, because score distributions vary enormously between queries and a global normalisation will simply encode the average query.
Pick weights empirically, and do it as a sweep rather than an argument. Take your evaluation set, run it at dense weights from 0.3 to 0.9 in steps of 0.1, and plot recall@k for each. The curve is usually flat in the middle with a clear cliff at one end. Do the sweep per query type as well as in aggregate, because the optimum for identifier queries and the optimum for conceptual questions are genuinely different, and that divergence is the strongest argument for routing queries to different weights based on a cheap classifier.
Retrieve Wide, Then Rerank: Depth and What It Costs
The dominant production pattern is two-stage. Stage one is cheap and recall-oriented: run both retrievers, fuse, and take a wide candidate set. Stage two is expensive and precision-oriented: score every candidate against the query with a much stronger model, and keep only the top few for the context window. This works because the first stage only has to get the right document somewhere in the candidate set, which is a far easier problem than getting it into the top five, and the second stage only has to sort a small list.
How wide should the candidate set be? The honest answer is that you measure it: plot recall@k against k for your fused retriever and find where the curve flattens. In most enterprise corpora that happens somewhere between 50 and 200 candidates. Below 50 you are usually still leaving recall on the table; above 200 the additional documents are almost never relevant and you are paying reranking cost for nothing. If recall@200 is barely better than recall@50, going deeper cannot help you, and the bottleneck is upstream in parsing, chunking or the fusion weights.
Depth costs latency, and this is where architecture meets the product requirement. First-stage retrieval over a well-tuned index is typically 10 to 50 milliseconds. Reranking a candidate set of 50 with a cross-encoder adds roughly 100 to 400 milliseconds depending on model size, sequence length and whether you are on GPU or CPU, and the cost scales close to linearly with candidate count, so a top-200 rerank is a several-hundred-millisecond to low-second addition. For an interactive assistant with a sub-second target before the first token, top-50 reranking is usually affordable and top-200 usually is not.
Two techniques buy back most of that. Batch the candidate scoring rather than issuing sequential calls, which on GPU turns a linear cost into something much closer to constant up to the batch size. And cascade: rerank the top 200 with a small, fast model, keep the top 40, then rerank those with the expensive model. A cascade routinely delivers most of the quality of deep reranking at a fraction of the latency. Whichever you choose, measure p95 rather than the mean, because reranking latency has a long tail that averages hide.
Cross-Encoders versus Bi-Encoders
The reason reranking works at all is architectural. A bi-encoder, which is what your vector index uses, embeds the query and the document independently and compares the two vectors. That independence is what makes it fast, because every document can be embedded once at ingestion and searched with a nearest-neighbour lookup. It is also what makes it imprecise: the document vector was produced without any knowledge of the query, so it has to be a general-purpose summary of the passage rather than an answer to a specific question. Speed is bought with that independence.
A cross-encoder puts the query and the document through the model together, so every layer can attend across both. It can therefore see that the query's negation applies to the document's claim, that the identifier in the query matches the one in the third line of the passage, and that a passage which looks topically perfect actually answers a different question. That joint attention is why cross-encoders outperform bi-encoders substantially on ranking quality. It is also why they cannot be used for retrieval: scoring requires a forward pass per query-document pair.
That constraint dictates the architecture. Bi-encoders search, cross-encoders sort, and the candidate set is the interface between them. When you choose a reranker, the axes that matter are the maximum sequence length, which must comfortably exceed your chunk size or the model silently truncates the end of every chunk, multilingual coverage if your corpus is not purely English, and throughput at your candidate depth. Turkish coverage in particular varies widely between reranker models and is worth testing directly on your own evaluation set rather than assuming from a model card.
There is a third option worth knowing: a listwise reranker, where a language model is shown the query and a numbered list of candidates and asked to return them in order. It can be very strong, especially with a capable model, and it needs no specialised reranking model. It is also slower, more expensive, non-deterministic unless you pin temperature to zero. We reach for it when the candidate set is small, when quality dominates latency, or when a domain-specific ordering rule is easier to express in a prompt than to train.
Query Understanding and What It Costs
Everything above assumes the query is a reasonable search string. Often it is not. Users type fragments, they type follow-up questions that only make sense given the previous turn, they bundle three questions into one sentence, and they use internal shorthand the corpus spells out in full. Query understanding is the set of transformations that sit between the user's text and the retrievers, and it is frequently the highest-leverage part of the whole pipeline because it fixes problems that no amount of index tuning can reach. It is also the part most teams add last.
Rewriting is the cheapest and most valuable. In a conversational interface, resolving pronouns and ellipsis against the conversation history before retrieving is close to mandatory. Decomposition splits a compound question into sub-questions retrieved separately, which is what makes multi-hop answers possible at all. Multi-query expansion generates three to five paraphrases and unions their results, trading three to five times the retrieval cost for a meaningful recall gain on vaguely phrased questions. HyDE goes further and asks the model to write a hypothetical answer, then embeds that answer instead of the question, because an answer looks more like an answer than a question does.
All of these cost a language model call before retrieval even starts, which is typically 200 to 600 milliseconds and a token bill on every single query. Our rule is to apply them selectively: always rewrite in conversational contexts because it is nearly free relative to its value, use decomposition only when a cheap classifier detects a compound question, and treat multi-query and HyDE as opt-in for query classes where you have measured a genuine gain. Applied unconditionally they can also hurt, because expansion introduces terms the user never asked about and pulls in confidently irrelevant results.
One security note. Query rewriting means user text is fed into a model whose output then drives retrieval, and retrieved documents are fed into a model whose output drives the answer. Both are injection surfaces, and a rewriting step that can be manipulated into changing filters or expanding scope is a real access-control risk rather than a theoretical one. The relevant attack patterns and mitigations are covered in AI security and prompt injection; the minimum here is that a rewriter must never be able to alter the access-control filters applied to the search.
Filtering and Access Control: Before or After Ranking
Most real deployments need filters: this department, this date range, this document type, this permission group. Where you apply them changes both correctness and quality. Post-filtering retrieves the top k and then removes anything the user may not see, which is simple to build and quietly broken. If four of the top five results are filtered out, the user gets an answer built from one document with no indication that the evidence base was gutted. It also means the unfiltered candidate list existed in memory and probably in a log, which is a disclosure risk on its own.
Pre-filtering restricts the search space before ranking so that k results are always k permitted results. This is correct, and it is what you want, but it interacts badly with approximate nearest-neighbour indexes: a graph index like HNSW navigates by proximity, and if most of the graph is excluded by a filter, the traversal wanders through ineligible nodes and recall collapses. This is why highly selective filters, the ones that eliminate more than roughly 95 percent of the corpus, are the classic failure case for naive pre-filtering. Selectivity is therefore the number to measure first.
The practical answers depend on selectivity. For low-selectivity filters, filtered graph traversal as implemented in engines such as Qdrant, or a filtered index scan in pgvector, works well. For high-selectivity filters, partition instead: separate collections or namespaces per tenant or per security domain, so the filter becomes a routing decision rather than a search constraint. For access control specifically, we favour partitioning at the tenant boundary and indexed label filters inside it, plus a permission-differential test set where the same question is asked by users with different rights and the answers are expected to differ.
Index Tuning: HNSW Parameters and the Recall Trade
Approximate nearest-neighbour search is approximate on purpose, and the amount of approximation is a parameter you own. HNSW, the graph index behind most vector databases including pgvector, Qdrant and the vector features of Elasticsearch and OpenSearch, has three settings that matter. M controls how many neighbours each node keeps, typically 16 to 64, and it sets the memory footprint and the quality ceiling. Higher M means better recall and a larger index. Once the index is built, M cannot be changed without rebuilding. Choose it before the first large index build.
efConstruction controls how hard the builder searches when inserting each node, typically 100 to 500. Raising it makes index construction slower but produces a better graph, and since construction happens once, it is usually worth setting generously. efSearch is the one you tune at query time: it controls how many candidates the traversal keeps in flight, typically 40 to 400, and it trades latency directly for recall. The important property is that efSearch is a runtime parameter, so you can measure the curve on your own data and change your mind later without rebuilding anything.
The tuning procedure is mechanical and takes an afternoon. Fix M and efConstruction, then sweep efSearch and plot recall against latency at each value, where recall here means agreement with an exhaustive exact search over the same corpus, not answer quality. The curve rises steeply and then flattens, and the correct setting is just past the knee. Teams that skip this almost always run defaults that either waste latency for recall they cannot use or, more commonly, silently lose several points of retrieval quality that no downstream reranker can recover.
Two related decisions travel with this. Quantisation, whether scalar or product, shrinks vectors substantially and speeds search at some cost in precision; it is usually a good trade above a few million vectors, especially when a reranking stage can repair the ordering afterwards. And the distance metric must match how the embedding model was trained, which for most modern models means cosine or inner product on normalised vectors. A mismatch here does not throw an error. It just quietly returns worse results forever, which is why it belongs on the same checklist as the analyser configuration.
Tune Against the Evaluation Set, Not Intuition
Everything in this article is a knob, and there are now more knobs than a team can reason about: fusion method, fusion weights, candidate depth, reranker choice, top-n into context, query transformations, filter strategy, efSearch, chunk size. The number of combinations is large enough that intuition is worthless, and the only way through is measurement. That means the evaluation set described in RAG evaluation metrics is not an optional companion to this work, it is the precondition for it. Without it, every change in this article is a guess with a plausible story attached.
Tune in a fixed order, because the parameters interact and a random order wastes time. First fix parsing and chunking, since they determine what can be retrieved at all. Then tune first-stage retrieval, sweeping fusion weights and candidate depth against recall at your candidate depth, ignoring the final answer entirely. Then add and tune the reranker, measuring recall and NDCG at your final context depth. Then tune query transformations. Then, and only then, work on the prompt. Moving to the next stage before the previous one has plateaued means you will re-tune everything later.
Measure the cost of every quality gain in the same table as the gain itself. A configuration that lifts recall@5 by three points while adding 400 milliseconds at p95 and doubling per-query cost is a legitimate choice for a due-diligence workflow and an obviously bad one for a customer-facing chat widget. We keep a single sheet with recall@5, NDCG@5, faithfulness, p50 and p95 latency and cost per thousand queries for every configuration tested, and the decision is made by reading a row rather than by relitigating an architectural preference in a meeting.
How We Build Hybrid Retrieval at HatsonTech
Our default starting architecture is deliberately unadventurous: BM25 with a properly configured Turkish analyser, dense retrieval over a chunked corpus, reciprocal rank fusion between them, a cross-encoder reranker over the top 50 to 100 candidates, and five to eight chunks into the context window with metadata filters applied before ranking. That configuration is not clever and it is not new, and it outperforms most of the pure-vector systems we are asked to review. We start there precisely because it is boring, then we tune it against the evaluation set and only add components that earn their latency.
The recurring findings are worth stating plainly. Adding a lexical leg to a dense-only system produces the largest single jump we see in Turkish enterprise corpora, and almost all of that jump lives in identifier and rare-term queries that were previously returning nothing usable. Reranking a wider candidate set beats upgrading the generator. And a badly configured analyser or a default efSearch has silently capped more Turkish search deployments than any model choice we have encountered. None of that is exotic engineering; it is a checklist that somebody has to actually run.
In the work behind caseon.ai and DiligenceAI, retrieval quality is measured as a first-class product metric rather than inferred from user complaints, and configuration changes ship through the same regression gates as code. If you are running a semantic search system that is good at paraphrase and bad at reference numbers, that is a solvable and well-understood problem, and it is usually a matter of weeks rather than months. Our RAG and semantic search practice starts by measuring where your current retriever loses, then fixes the stages in the order that pays.