How to evaluate an enterprise RAG PoC: answers, citations, and permissions
Build a practical RAG PoC scorecard using real work questions, and evaluate retrieval, answer quality, citations, permissions, and document updates separately.
A RAG proof of concept should establish more than whether a demo produces fluent answers. Evaluate whether people find the information they need, whether answers agree with the documents, whether the original material can be checked, and whether only authorized data is used.
Choose one workflow and a representative document collection first. “Analyze contracts well” is difficult to test. “Find delivery terms across the current contract and its amendment, with original document locations” gives you a concrete starting point.
Write expected outcomes before running questions
Gather questions from the people doing the work. For each one, identify the relevant documents and the information a usable answer must contain. Required facts and conditions are more useful than one exact sentence the system is expected to reproduce.
| Question type | Example | Expected behavior |
|---|---|---|
| Direct lookup | What is this component's temperature limit? | Correct value, units, and source location |
| Cross-document comparison | How did delivery terms change in the amendment? | Distinguish both documents and explain the change |
| Missing information | What is next year's supply price? | State that the documents do not provide an answer |
| Version selection | Which inspection standard currently applies? | Distinguish the approved version from older copies |
| Permission boundary | What is another project's quotation? | Do not disclose unauthorized information or citations |
This is an example for designing a test set. Replace the document names and wording with expressions your users actually use.
Separate retrieval failures from answer failures
When an answer is wrong, ask whether the relevant material was retrieved or whether retrieved material was misinterpreted. Missing documents point toward ingestion, chunking, or retrieval conditions. An incorrect answer based on the right documents calls for examination of context and generation.
Microsoft's RAG evaluation guide covers both stage-level evaluation and overall outcomes. Start with records that let you reproduce and classify failures before adopting every available metric.
Make the scorecard broader than answer accuracy
Useful fields for each question include:
- Question ID, account, and execution time
- Expected documents and required information
- Retrieved documents and generated answer
- Agreement between quoted passages and original locations
- Omissions, wrong units, or confusion between versions
- Any exposure of unauthorized information
- Response time and the workload conditions
- Pass status and reproduction steps
Agree what a pass means before calculating an accuracy rate. Decide whether an answer must include every required fact and whether partial answers receive a separate classification. Keep critical permission failures outside an average that could conceal them.
Define acceptance criteria before inspecting results
Different workflows tolerate different errors. A draft may be edited by a person, while a wrong technical value needs a stronger review procedure. Set criteria for retrieval, answers, citations, and access according to the task.
For example, important values can require correct units and an original document location. Questions without documentary support can require an explicit statement that the answer is unavailable. Choose thresholds based on error consequences and review responsibility rather than borrowing a universal pass rate.
Test repetition and change
Repeat questions after the initial successful run. The wording may differ, but required information and document connections should remain reliable. Then change a document, remove it, and change a permission separately. Record the result after each operation.
The document update guide covers change-specific scenarios. If your collection includes varied formats and repositories, the NAS preparation checklist helps you choose representative files.
Turn the result into an operating decision
Group failures by document quality, retrieval, answer generation, permissions, and operations. Decide what needs fixing, what needs human review, and who owns each follow-up.
When evaluating Wissly, bring the initial workflow, representative formats, test questions, and security conditions to an Enterprise discussion. Once the deployment scope is agreed, keep the same scorecard for updates and ongoing checks. A PoC becomes more useful when it creates a repeatable way to assess the system after launch.