RAG is an AI architecture in which a system first searches a separate knowledge base for relevant information and then gives those passages to a generative model. The abbreviation means retrieval-augmented generation. The model therefore does not need to know your manuals, contracts or product information from its training memory.
RAG is useful when knowledge is spread across documents and changes regularly. It is not a truth machine. If the system retrieves the wrong passage, finds an outdated version or misreads an exception, the answer can still be convincingly wrong.
When RAG fits better than an ordinary chatbot
RAG is worth investigating when:
- employees repeatedly answer the same questions from many documents;
- answers must point to a verifiable source;
- knowledge changes without retraining a model;
- users may search only their own files or department;
- natural-language questions are longer and more varied than keywords.
Prefer ordinary search filters when there are few documents, exact terms matter or users must read the original document. Use deterministic software when rules fully determine the answer. Fine-tuning can adjust style or task behaviour, but it is not a simple replacement for a current, citable knowledge base.
The chain in six steps
- Ingest: collect documents from SharePoint, Drive, a CMS, case files or other approved sources.
- Clean: identify duplicates, expired files and unreadable content, then add metadata.
- Chunk and index: split documents into useful passages and make them searchable.
- Retrieve: find likely relevant passages for a question and optionally rerank them.
- Generate: have the model answer within the retrieved context and instructions.
- Show and log: retain source, version, answer, user and feedback for review.
A polished chat interface is only the final layer. Source management, permissions and retrieval usually determine quality more than prompt wording.
Start with source quality
Create a source register before the technical pilot.
| Field | Why it is needed |
|---|---|
| Owner | Who decides whether the document is valid? |
| Version and validity date | Which passage wins when sources conflict? |
| Confidentiality | Who may see the source and derived answers? |
| Document type and language | How should the text be read and divided? |
| Relationship to other sources | Is this main policy, appendix, local exception or archive? |
| Removal and update process | How does outdated knowledge leave the index? |
If two procedures conflict, RAG can expose the conflict but cannot decide which policy is institutionally valid. That needs an owner.
Enforce access rights before retrieval
Do not filter only after the model has produced an answer. The search layer may retrieve only passages the user could open without AI. Otherwise an answer can leak details from a document that is not displayed as a citation.
At minimum, test these cases:
- employee A asks about employee B's file;
- a user crafts a prompt intended to summarise a hidden source;
- a public document links to a confidential appendix;
- withdrawn access remains available through a cache or log;
- a document contains instructions pretending to be a system command.
The last case is indirect prompt injection. Treat document text as data, never as a change to system policy.
Build an evaluation set before the demo
Collect fifty to one hundred real questions for a first defined use case. For each question, record the expected core answer, required passages, allowed sources per role, important exceptions and the required behaviour when evidence is missing.
Measure retrieval and answer quality separately. Were the required passages among the top results? Did the answer stay within those sources, include the right exception and cite verifiably? A well-written answer based on the wrong source is not a success.
Citations are necessary but not sufficient
A citation proves that a passage was used, not that it is current or correct, nor that the conclusion follows. Let users open the exact passage and its document metadata. Mark missing evidence clearly, and do not fill gaps from general model memory when the application permits only internal sources.
Expert review remains necessary for legal, financial or safety-critical advice.
The real cost items
The model call is often not the largest cost. Budget for integrations and synchronisation, OCR for scans and tables, storage and retrieval, identity management, evaluations, editorial ownership, logging, security and incident response.
A pilot with thirty clean PDFs says little about production with tens of thousands of versions, attachments and permissions. Test early with messy reality.
A safe pilot architecture
Limit the first version to one department, one source set and read-only answers. Always show sources and allow “insufficient evidence” as a valid outcome. Give users separate ways to report a wrong answer, wrong source and missing knowledge.
Move to actions only after the assistant demonstrably retrieves reliably. First read what an AI chatbot can and cannot do and complete the AI-readiness check.
Go or no-go before production
- Every source has an owner and validity status.
- Document permissions are enforced in retrieval.
- The evaluation set contains real exceptions.
- Source passages can be opened directly.
- Missing evidence causes refusal or escalation.
- Updates and deletions demonstrably reach the index.
- Cost, latency and error rate stay within chosen limits.
- A process owner follows up feedback and incidents.
RAG makes company knowledge easier to reach when source management is already taken seriously. Without that foundation, it mainly produces faster answers from the same mess employees already distrust.