Why retrieval misses the answer that is there

When an assistant cannot answer a question your documents already answer, the model is rarely the problem. The passage was never put in front of it. Six things cause that, they have different fixes, and the first job is to find out which one you have - by measuring retrieval separately from the answer. The single most common missing piece is lexical search alongside the vector search.

“The AI does not know our own documentation” is a retrieval report, not a model report.

Key takeaways

Measure whether the right passage was retrieved before you touch the prompt or the model.
Vector search alone fails on exact strings: error codes, SKUs, part numbers, policy references.
Adding keyword search next to it fixes more real misses than any amount of embedding tuning.
Recall is the ceiling. Nothing downstream can recover a passage that was never a candidate.
A miss that reaches the user as a confident wrong answer is worse than one that reaches them as “I do not know”.

Who this applies to

You have a working assistant over your own content and it fails on questions you can answer yourself in thirty seconds with the search box. This is about diagnosing that specific failure.

If the question is how to structure long documents in the first place, that is a design question rather than a debugging one, and it is covered in context windows and why long documents still fail. This article assumes the system is built and something is wrong with it.

Prove it is a retrieval miss first

There are two entirely different failures behind the same complaint, and they need opposite fixes.

The passage was never retrieved. Nothing downstream can recover from this. The model answered from whatever it was given, which did not contain the answer.

The passage was retrieved and the answer is still wrong. Now it is a prompt, a reranking or a model problem, and it is a much smaller category than teams assume.

The test is mechanical. Take twenty failing questions, find by hand the passage that answers each, then check whether that passage appeared in what was sent to the model. Two afternoons of this is worth more than a month of prompt edits, and in most systems we have looked at, the great majority land in the first category.

That check is retrieval hit rate, and it belongs in your evaluation set as its own number rather than as a component of answer quality. The method for building the set is the same one used for an eval set from real tickets; the only addition is recording, for each case, which document and section should have been found.

The narrowing is the point: each stage passes on less than it received. Diagnose from the top, because a fix at gate four cannot recover what gate one dropped.

The six reasons a passage does not arrive

Each has a different test and a different fix. Work down the list, because the cheap causes are also the common ones.

#	Cause	The test	The fix
1	It was never indexed	Search the index directly for a phrase you know is in the document	Fix the pipeline: failed parses, scanned pages, indexing lag
2	The words do not match	Compare the user’s phrasing with the document’s	Hybrid search, and query rewriting
3	It contains an exact string	Search for the code or SKU on its own	Lexical search; vectors are the wrong tool for identifiers
4	A filter removed it	Re-run the query with filters off	Correct the metadata, or the filter logic
5	It was a candidate but ranked too low	Raise the candidate count and look further down	More candidates, then rerank
6	The answer spans two chunks	Read the chunk boundaries around the answer	Overlap, or section-level retrieval

Cause 1 is embarrassing and frequent. A scanned PDF with no text layer, a parser that silently produced empty content, a document added after the last index run. Nobody checks, because the assumption is that indexing worked. Search your index for a literal sentence you can see in the source file; if it does not come back, everything after this is wasted effort.

Cause 4 is the quiet one. A permission filter, a date filter, or a “current version only” flag is doing exactly what it was told and excluding the answer. Running the same query with filters disabled separates a retrieval problem from a metadata problem in one step. If the filter is a permission boundary, though, the correct outcome may genuinely be a refusal - see RAG permissions.

The vocabulary gap, and the identifier problem

Two of the six deserve more than a table row, because together they are the majority of real misses.

Users and documents use different words. A customer writes “can I get my money back”. The policy says “reimbursement is available within the stated period subject to condition of goods”. These are semantically close, which is what embeddings are good at, and the gap is often still enough to push the right passage below the cut. Internal documentation is worse than customer-facing content here, because it is written in the vocabulary of the department that owns it.

Exact strings are where vector search is genuinely weak. An error code, a SKU, a part number, a clause reference like “7.3(b)”, an API parameter name. These carry almost no semantic signal. Embedded, ERR_4021 sits near every other error code, and the nearest neighbours are confidently wrong. A keyword index finds it immediately, because it is a string match and it was never a semantics problem.

This is why hybrid search matters more than any other single change. Run the query through both a vector index and a keyword index, merge the two candidate lists, then rerank the merged set. Each method covers the other’s blind spot: the vector side handles paraphrase, the lexical side handles identifiers and rare terms.

If your system does semantic search only, and your domain has any codes, names or references in it, this is almost certainly the largest available improvement, and it is a smaller piece of work than migrating an embedding model.

Neither column is better. They fail on different questions, which is the whole argument for running both rather than choosing.

Query rewriting, and where it backfires

The second fix for the vocabulary gap is to change the query rather than the index. Three variants, in increasing order of risk:

Expansion with synonyms from your own domain. A short mapping from customer words to document words. Unglamorous, cheap, and it fixes a specific measured problem.

Resolving the conversation. “What about the other one?” is not a searchable query. Rewriting it to include the referent from earlier turns is usually a clear win, and it is a common omission in multi-turn assistants.

Model-generated rewrites and hypothetical answers. Have a model rewrite the question, or draft a hypothetical answer and search with that. Sometimes it helps considerably. It also adds a step that can silently make retrieval worse, costs latency on every request, and introduces a second thing that changes when your provider updates a model.

The rule for all three: it is a change like any other, so it needs a before number and a measured after. A rewriting layer added on intuition is a common source of a system that got worse for reasons nobody can locate.

Recall is the ceiling, and it is measured differently

The metric to fix on is recall at the candidate stage: of the questions in your set, for how many was the correct passage among the candidates retrieved, before reranking.

That number is a hard ceiling. Reranking can reorder candidates, a better prompt can use them better, a stronger model can reason over them more carefully. None of them can use a passage that was never a candidate.

So the sequence is: get recall high with a generous candidate count and hybrid search, then use reranking to cut that list down to what actually goes in the prompt. Retrieve twenty, rerank, send three to five. Teams frequently do the opposite, retrieving five and tuning the prompt, which optimises precision inside a candidate set that is already missing the answer.

One caveat that keeps this honest: a bigger candidate list is not free. It costs a reranking pass and it costs latency, which is a real budget rather than a rounding error - see latency budgets for conversational AI. The point is that recall is the thing to buy first, not the thing to buy without limit.

The three lower boxes are where most tuning effort goes. All of it operates below a line somebody else set when the candidate list was built.

What we default to

Hybrid retrieval from the start, on every knowledge system, even when the client’s content looks like plain prose. It has never been the wrong call, and the cost of adding it later - reindexing, rewriting the query layer, re-measuring - is several times the cost of having it from the beginning.

Retrieval hit rate as a named metric in the evaluation report, separate from answer quality, so that a bad month can be attributed to one half or the other without an investigation.

And a system that says it does not know, rather than answering from the best of a bad candidate set. This is the recommendation clients push back on most, because a refusal looks like a failure on a dashboard and a confident wrong answer does not. The argument that lands is a specific one: a wrong answer about a returns window creates a complaint, a refund and a support contact, and the assistant is now generating work rather than removing it. That trade is the subject of what hallucination rate is acceptable.

Where we have been wrong: we spent too long treating misses as an embedding-model choice. Changing the model gave small, hard-to-measure movements. Adding a keyword index alongside it, on the same corpus, fixed a whole category of failure immediately. The lesson we took is that the cheap structural fix should be exhausted before the fashionable one.

When this is not worth it

When the corpus is small enough to send whole. A dozen short policy documents may not need retrieval at all. Diagnosing retrieval you should not have built is wasted work.

When the content genuinely does not contain the answer. Some of what reaches an assistant was never written down anywhere. The fix is a documentation project, and no retrieval work substitutes for it. This is worth checking early, because it is common and it changes who owns the problem.

When the questions are about the whole document rather than a part of it. “Summarise the contract” is not a retrieval failure when it comes back thin. It is the wrong design, and the alternative is in context windows and why long documents still fail.

Frequently asked questions

Will a better embedding model fix our misses?

Occasionally, and less often than the release notes suggest. Test it the same way as anything else: same corpus, same set, recall measured before and after. In our experience adding lexical search moves the number more than changing the embedding model does, and it is cheaper to do.

How many candidates should we retrieve?

Enough that recall stops improving, then rerank down. Twenty to fifty candidates before reranking is a common working range; the right number is the one your own measurement shows, and it depends far more on chunk size than on corpus size.

Is a reranker worth the extra call?

Usually yes, once you are retrieving enough candidates for recall. It is what makes a generous candidate list affordable in the prompt. It is not a fix for low recall, because it can only reorder what it was given.

Do we need a graph or a knowledge base instead?

Only for questions that require traversing relationships, such as “which contracts inherit terms from this master agreement”. For question answering over documents, hybrid retrieval plus reranking handles the great majority, and a graph is a significant amount of modelling work to maintain.

How do we know whether it is retrieval or the model?

Take the failing question, put the correct passage in the prompt by hand, and ask again. If the answer is now right, it was retrieval. If it is still wrong, it is the prompt or the model. That single test resolves most arguments about this in a few minutes.

Next step

If your assistant is failing on questions your documents answer, the measurement is a day and it tells you which of six problems you have. The knowledge assistant engagement starts with retrieval hit rate on your own content, before any prompt work.