Five steps to diagnose low RAG search accuracy
Diagnose RAG errors across extraction, chunking, hybrid search, reranking, and answer grounding. Use the same questions to locate failures and compare changes.
When RAG search accuracy is low, find where the required evidence disappears before changing the model. The source may exist but fail extraction; it may be missing from search candidates; it may appear among candidates but be excluded from the final context; or the model may produce a wrong answer despite receiving the right evidence.
Similar wrong answers can have different causes. Changing an embedding model does not repair a broken table extraction, and generating a longer response does not fix the use of an incorrect document revision. Work through collection and extraction, chunking, search, ranking, and answer generation to narrow down what needs to change.
Pair each wrong answer with the evidence it should have used
For every question under investigation, identify the source file and precise evidence location. Record the filename, document version, account used, question, execution time, and answer. Reproduction conditions matter because different settings or document states can produce different results for the same question.
Describe the observed failure instead of writing only “low accuracy.” This makes an investigation easier to continue when another person takes over. Use the following table to choose where to look first.
| Observed result | First record to inspect | Initial focus |
|---|---|---|
| File appears, but a sentence in its body cannot be found | Extracted text and processing status | Collection and document extraction |
| Body text is present, but conditions or footnotes are missing | Retrieved document chunks | Chunking and context preservation |
| A similar product appears instead of the requested item code | Search query and candidate documents | Exact strings and retrieval method |
| Correct evidence appears only near the bottom of the candidates | Lists before and after ranking | Reranking and result selection |
| Correct context is supplied, but the answer is wrong | Evidence sent to the model and its answer | Generation, citations, and abstention |
Microsoft's RAG design and evaluation guide also distinguishes retrieval choices and evaluation at individual processing stages. Even if your organization does not build the system itself, you can ask a vendor to report findings using the same distinctions.
Step 1. Confirm the evidence is present in the extracted text and index
Find the answer in the original, then compare it with what the search system actually read. A document registration indicator does not prove that its entire contents entered the searchable collection. Processing may have covered only some pages or excluded tables, images, or password-protected files.
Check whether the required sentences, table values, units, and exceptions survive extraction. For scanned PDFs, compare numbers against the OCR output. For multi-column documents, check reading order. Address these problems during OCR and PDF indexing validation first.
If the text was extracted but cannot be retrieved, inspect index status, the selected collection, update timing, and filter conditions. A result excluded because the user lacks access is different from a relevance failure. Use an authorized test account that can read the document instead of relaxing production permissions to diagnose the problem.
Keep a more specific record than “file processed successfully.” Link the original evidence passage, extracted passage, indexed document ID, and processing time. This allows a direct comparison after the file is processed again.
Step 2. Check whether chunking separates essential conditions
Chunking divides long documents into pieces that can be retrieved. Very small chunks may lose surrounding conditions, while a chunk combining unrelated material can make the relevant passage harder to find. The appropriate size depends on the document structure and the questions being asked.
Microsoft's document chunking guidance describes fixed-size, structural, semantic, and combined approaches. Rather than treating one fixed length as the right setting for every document, compare whether the necessary context survives in the files that currently fail.
For example, a section heading defining who a policy covers, an expense table, and a footnote describing approval exceptions may become separate chunks. Search could then retrieve only the table's numbers. Test retaining headings and document information with chunks, or retrieving relevant neighboring content. Adding context can also introduce irrelevant passages, so evaluate the resulting answers again.
| Document structure | Context to check | Question to rerun after changes |
|---|---|---|
| Policy with exception clauses | Connection between the main rule and exception | “Does the same rule apply in this situation?” |
| Table spanning multiple pages | Column headings, rows, and units | “What is the value and unit for this item?” |
| Long manual | Chapter or section heading and applicable equipment | “Which model does this procedure apply to?” |
| Body text referring to an appendix | Relationship between the reference and appendix | “What are the detailed conditions mentioned in the body?” |
After changing chunking, rebuild the affected index entries before comparing results. Changing several other settings at the same time makes the cause of an improvement difficult to identify. Start with one change against the same documents and questions.
Step 3. Test exact strings and semantic retrieval separately
Internal questions often contain document numbers, equipment codes, or customer abbreviations that need exact matching. Other questions use different wording from the source: an employee might ask why a machine “keeps stopping,” while the manual describes “recurring shutdowns.” Evaluating both only as one combined question set can hide a weakness in either category.
Keyword search can help retrieve exact codes and names. Vector search can retrieve material with a similar meaning despite differences in wording. Hybrid search combines the two result sets to address these needs together. The Azure AI Search overview of hybrid retrieval also distinguishes specialized terms and codes from semantic similarity.
Enabling hybrid search is not evidence that accuracy has improved. Adapt these hypothetical test questions to your own terminology and compare the results:
- Exact identifier: “What is the inspection interval for equipment code EQ-17?”
- Different wording: “What should we check when the equipment repeatedly stops?”
- Combined conditions: “What should be inspected during high-temperature operation of EQ-17?”
- Missing evidence: “What is the inspection interval for the new EQ-99 model, which is not in the documents?”
Compare keyword, vector, and combined retrieval against the same material, recording whether the correct document appears among the candidates. Check how hyphens, numbers, and abbreviations are handled, and whether identifiers and conditions survive query rewriting. Exact-match and semantic retrieval can help with different types of questions.
Step 4. Find evidence lost between candidates and the final context
If the correct document is an initial candidate but absent from the evidence the model receives, inspect result selection. Check candidate counts, duplicate documents, reranking, and context-length limits in sequence. Assuming that all search results are passed to the model hides this distinction.
Reranking compares retrieved candidates with the question again to adjust their order. Microsoft's semantic ranking overview describes it as an additional ranking stage over an initial result set. Reordering candidates alone cannot recover evidence that never entered that set.
Retrieving more candidates may increase the chance of including the right evidence, but it can also change processing time and cost and introduce irrelevant material. Record candidate counts and final evidence counts separately for the same question. Check whether duplicate copies crowd the top results.
Keep access and document-version requirements separate from relevance scores. A highly relevant old policy may still be the wrong source for a question about current rules. Define how approval status and effective dates are applied, and repeat document update checks after revisions or deletions.
Step 5. Check whether answers fail despite receiving the right evidence
If the model receives the required evidence but answers incorrectly, investigate what happens after retrieval. Separate misread numbers or units, omitted exceptions, conditions combined across unrelated documents, and explanations unsupported by the source. At this point, comparing changes to answer instructions, model choice, and context assembly can be useful.
Define an answer format suited to the workflow. Separating the main conclusion, applicable conditions, source documents and locations, and items requiring confirmation can make review easier. A tidy format does not establish correctness, so compare each claim with the original.
Include questions the documents cannot answer. Even when related material exists, check whether the system explains that the information needed for a conclusion is missing. A source link is not a passing result if the actual passage does not support the answer.
Unauthorized information appearing in answers, titles, or citations should be handled separately from average answer accuracy. Include both access checks before evidence reaches the model and behavior after access revocation in your enterprise AI security review.
Compare changes against the same criteria
Keep questions, document versions, test accounts, and acceptance criteria consistent before and after a change. Rerun questions that previously passed as well as those you fixed. Holding back questions that were not used during tuning can help reveal whether a change works only for the examples used to develop it.
Record evidence retrieval, rank, inclusion in the final context, answer conditions and values, citation agreement, and response time separately. An improvement on a few questions does not establish an overall accuracy gain. Report the document types and question scope covered by the test alongside the results.
Frequently asked questions
Will switching embedding models immediately improve retrieval accuracy?
It may help when the model is the cause, but extraction gaps, inappropriate chunking, overly restrictive filters, and failed updates need separate fixes. When changing a model, check compatibility between indexed and query embeddings and identify what must be reindexed. Compare results using the same questions.
Does supplying more search results make answers more accurate?
Not always. More results may add necessary evidence, but they may also introduce duplicates, irrelevant documents, or conflicting versions. Adjust search candidate counts separately from the evidence supplied to the model, and assess accuracy, response time, and processing cost together.
Does buying a solution remove the need for these checks?
Document structures, terminology, and permissions vary by organization, so representative files still need testing. Even if you cannot adjust the internals directly, you can provide failed questions and expected evidence and ask the vendor to explain which stage caused the problem.
What to prepare when evaluating Wissly
Wissly supports searching connected internal documents and checking the original evidence behind answers. Evaluate whether it finds the information your organization's documents and questions require, beyond the name of a particular search technique. Retrieval, document processing, and integrations are reviewed in the context of the intended deployment.
Bring failed questions, expected source locations, file formats, storage systems, and security requirements to a Wissly deployment consultation to define a concrete evaluation scope. Start by separating the task of recovering evidence that search missed from the task of correcting an answer that misinterpreted the evidence it received.
