RAG for enterprise AI teams: from the basic idea to everyday work

When I meet people responsible for AI adoption in their companies, many are learning the terminology as they go. Some have a strong technical background. Others have just been asked to lead an AI initiative, and terms such as RAG and embeddings are still unfamiliar.

Searching for an explanation of RAG often takes them straight to developer tools and implementation tutorials. I wanted to offer a starting point that does not require a development background. A set of meeting notes is enough.

Read the meeting notes, then write the email

Ask an AI to “write a confirmation email saying we agreed to build feature B for company A,” and it can draft the email from the facts in your request.

Now ask it to “find the meeting notes with company A and write a confirmation email.” It has to find the notes, read what was agreed, and use that information in the draft. That is the basic idea behind Retrieval-Augmented Generation: retrieve relevant material and use it to produce an answer or document.

If the notes say that feature B is agreed but the schedule will be discussed later, the email should reflect both points. It should not fill in a plausible delivery date. The connection between finding information and writing from it is what matters here.

The difficulties start with reading the document

The idea is simple. Working with actual company files is more complicated.

Extracting every character from a PDF table does not necessarily preserve its meaning. A list of names and researcher IDs becomes useless if the extraction loses which ID belongs to which person.

Splitting documents into smaller pieces creates a similar problem. These pieces, often called chunks, make material easier to retrieve. But a split based only on length might leave a name in one chunk and its ID in another. A model that receives only one of those pieces is missing part of the information it needs.

The system then has to find the right pieces for the question. Embeddings and vector search help find related meanings even when the wording differs. Exact names and identifiers also matter, which is why keyword search can be useful alongside semantic search. Decisions made while reading, splitting, and searching the documents all affect the answer.

A relevant result is not a complete list

“Find every company that contacted Wissly in the first half of the year and put them in a table.”

This sounds like a straightforward task for a document search system. But a search that selects only a few highly relevant passages can miss the point. With top-K retrieval, the model receives a limited number of results. Information about companies further down the ranking may never reach it.

The person asking wants every qualifying company, not a useful sample. Finding one fact accurately and producing a complete list require different approaches. We have repeatedly encountered this distinction while building Wissly.

Calculations present another challenge. Finding the figures is only part of the work. One approach is to have the model write calculation code, run that code in a separate environment, and explain the returned result. The calculation is carried out explicitly instead of relying on the language model to produce a plausible number.

“I cannot write it because it is not in the documents”

This was a particularly frustrating problem for us early on. To reduce invented facts, we instructed the AI not to produce information absent from the documents. That instruction made sense when the task was to look something up.

The problem appears when the task is to draft a new business plan. Of course, the finished plan does not already exist in the source material. Yet the AI can refuse to write because it cannot find the requested content. A request to write from company documents and the requirements in a call for proposals gets treated as a request to retrieve an existing, finished document.

We initially tried adding more conditions to the instructions: stay factual for lookup tasks, be creative for writing tasks. The behavior was inconsistent. One way to address this is to identify the intent of a request first and apply instructions suited to the task, such as lookup or drafting.

The source documents may be the same, but the work being asked of the AI is different. It needs to recognize that difference.

When the information becomes useful work

Our team uses development history from GitHub to draft research notes. The system reads the changes and uses them as evidence for the draft. We connected separate automation for submission and approval in IRIS, the research administration system. RAG handles the part that turns actual work records into a written account.

We also used it for customer support. Answering familiar questions meant repeating many of the same conversations. We organized previous support exchanges into FAQs and guides, then had AI retrieve the relevant material to answer questions. Information we already had became something we could use in everyday work.

Building this kind of system often means revisiting several stages to solve one problem. Document processing, retrieval scope, and writing instructions affect one another. We do not see RAG as a feature that is finished once it has been built. We keep improving it by looking at the requests people make and the results they receive.

That is what we are working toward with Wissly: making the information a company has accumulated easier for its people to use in their work. If you are considering what AI could do with your own documents, we would welcome a conversation.