Almost every company now has someone asking whether "we can put AI on our documents". The idea is appealing: a chatbot that answers questions from your policies, product manuals, contracts or support history, so people stop hunting through shared drives and interrupting colleagues.
The technique behind most of these systems is retrieval-augmented generation, or RAG. Instead of relying on what a language model learned in training, the system first searches your documents for relevant passages, then asks the model to answer using only those passages. Done well, it is genuinely useful. Done carelessly, it produces fluent, confident answers that happen to be wrong.
When it is worth it
A retrieval chatbot earns its keep when a few conditions line up:
- The knowledge exists in writing. If the answers live only in people's heads, no amount of retrieval will find them.
- The same questions recur. Support teams, sales engineers, onboarding staff and internal helpdesks are classic cases.
- The documents are reasonably current. A chatbot built on outdated material will spread outdated answers with great efficiency.
- A wrong answer is recoverable. Internal use, with sources shown and a human in the loop, is a much safer starting point than a customer-facing bot quoting prices or legal terms.
It is often not worth it when the underlying need is really search, when the corpus is tiny, or when answers depend on live data that sits in a database rather than in documents. In those cases, better search, a well-organised FAQ or a simple report may solve the problem faster and more reliably.
The unglamorous work that decides quality
Most of the effort in a good RAG system is not the model. It is the preparation of the material and the retrieval step. The questions that matter are practical:
- How are documents split? Chunks that are too small lose context; chunks that are too large dilute relevance. Respecting headings, tables and sections usually beats cutting at arbitrary lengths.
- What metadata travels with each chunk? Document title, date, owner, product and access level all improve retrieval and make filtering possible.
- How is retrieval done? Combining semantic search with plain keyword matching tends to handle product codes, names and jargon better than either alone.
- Who is allowed to see what? If a user cannot open a document in your drive, the chatbot must not quote it to them. Permissions have to be enforced at retrieval time, not bolted on later.
A retrieval chatbot is only as good as what it retrieves. Most failures that look like the model being wrong are really the search being wrong.
How to evaluate it honestly
Demos are persuasive because they use questions the builder already knows work. Real evaluation looks different. Before anything goes live, build a test set of genuine questions drawn from support tickets, emails or chat logs, along with the answer a knowledgeable person would give and the document it should come from.
Then measure a few things separately:
- Retrieval: did the system find the right passage at all?
- Faithfulness: did the answer stick to what the passage says, or embellish?
- Usefulness: would the person asking actually be helped?
- Refusal: when the answer is not in the documents, does it say so rather than improvise?
That last point is underrated. A system that says "I could not find this in the available documents" is far more trustworthy than one that always has an answer. Showing the source passages alongside every response lets users check for themselves and builds the right kind of confidence.
Keep the test set and rerun it whenever you change documents, prompts, models or retrieval settings. Without it, every change is a guess.
Start with a feasibility sprint
The sensible path is a short, bounded pilot: one well-defined corpus, one group of users, a test set agreed in advance, and a clear bar for success. At the end you know whether the approach works for your material, what it would cost to run, and what needs fixing in the documents themselves. Sometimes the most valuable outcome is discovering that the knowledge base needs tidying before any AI goes near it.
Notepoint's data and AI practice runs exactly this kind of engagement, from retrieval chatbot pilots with honest evaluation to broader ML feasibility studies and the data pipelines that feed them. If you are still weighing whether the idea fits your business, a short advisory session is a low-cost way to pressure-test it.
