The short answer
- Retrieval-augmented generation (RAG) searches your own documents for relevant passages and gives them to a language model with the question, so the answer is based on your sources.
- Hybrid search, which combines meaning-based vector search with keyword search, often finds the right passages more reliably than either method alone.
- Permissions must be applied when passages are retrieved, before anything reaches the model, so no customer or user sees documents they are not allowed to.
Retrieval-augmented generation (RAG) is a way of answering questions with a language model by first searching your own documents for the passages most relevant to the question, then giving those passages to the model with instructions to answer from them and cite them. It lets a model answer from information it was never trained on, such as your policies, contracts or product data, and it lets people check every answer against its source. RAG suits questions whose answer is written down somewhere in your documents, and it is the wrong tool for totals, calculations and tasks that need a whole document at once.
Why RAG exists
A language model knows what was in its training data and nothing about your business. It has not read your health and safety procedures, your price book or last year’s tender responses. Asked about them, it will either say it does not know or, worse, write a plausible answer that is wrong.
There are two ways to give a model your information. You can paste the documents into the prompt, which works when they are short. Or you can search them and send only the relevant parts, which is RAG. Once you have more than a handful of documents, searching is cheaper and faster, and usually more accurate, because the model is not distracted by thousands of pages that have nothing to do with the question.
How a RAG system works
- Ingest. Documents are collected from where they live, such as a file store, a document management system or the application’s own database, and turned into clean text.
- Chunk. Each document is split into passages small enough to be precise and large enough to make sense on their own, often a few hundred words. Good chunking follows the document’s headings and keeps the title and section name with each passage.
- Embed. Each chunk is passed through an embedding model, which turns text into a list of numbers called a vector. Texts with similar meanings get vectors that sit close together, even when they use different words.
- Index. The vectors are stored in an index along with the text, the source document, the page or section, and who is allowed to see it. The index can live in an ordinary database, such as PostgreSQL with the pgvector extension, or in a dedicated vector database.
- Retrieve. When someone asks a question, it is embedded in the same way and the closest chunks are found. Keyword search usually runs alongside, and a reranking step can reorder the candidates by how well they answer the question.
- Generate. The best few chunks go to the language model with the question and instructions to answer only from them, cite them, and say when the answer is not there.
Vector, keyword and hybrid search
Retrieval often matters more to the quality of a RAG system than the choice of model. If the right passage is not retrieved, the model cannot use it.
| Method | How it matches | Good at | Weak at |
|---|---|---|---|
| Keyword search, such as BM25 | Passages that contain the words in the question, weighted by how rare those words are | Exact terms, codes, part numbers, names and regulation references | Different wording for the same idea |
| Vector search | Passages whose meaning is close to the question | Paraphrases and questions in everyday language | Exact codes and rare terms, which can be lost in the overall meaning |
| Hybrid search | Runs both and merges the results | Real questions, which tend to mix everyday language with specific terms | Needs tuning, and is more to build and run |
| Reranking | A second model scores the top candidates against the question | Choosing the best few from a longer list | Adds time and cost to every question |
Business documents are full of exact terms. A question about “CDM 2015 duties of the principal contractor” needs keyword matching for “CDM 2015” and meaning-based matching for the rest, which is why we’d recommend hybrid search as the default.
Permissions and tenants
In business software, not everyone may see every document. In a SaaS product, each customer’s documents belong to that customer, and within a customer some documents are restricted to certain roles. A RAG system must respect both.
The rule is to filter at retrieval. Every chunk is stored with its tenant and access information, and every search includes a filter for the current tenant and user, applied before results are ranked and long before anything reaches the model. An instruction in the prompt telling the model not to reveal certain documents is not a security control, because a passage the model has been given can always end up in an answer. When permissions change, the index has to change with them, and a deleted document must be removed from the index as well as from storage. Our guide to multi-tenancy covers how each customer’s data is kept separate more generally.
Citations people can check
Every answer should show where it came from. Because each chunk carries its source document and location, the system can show a citation such as “Hot works procedure, section 4” beside each claim and link straight to that passage.
Citations also make errors visible. If the model cites a passage that does not support what it said, a reviewer can see the problem. You can check this automatically too, by confirming that each cited chunk was among those retrieved and that any quoted text appears in it. Where the documents do not contain an answer, the system should say so plainly. Designing for that case is covered in designing for wrong answers.
A worked example over a company’s documents
Take a building contractor with 2,000 documents in its shared drive, including health and safety policies, method statements, risk assessments, product data sheets and past tender responses. Say they average eight pages of 400 words each.
- That is 2,000 × 8 × 400 = 6.4 million words, or roughly 8.5 million tokens at about 1.33 tokens per word.
- Split into chunks of around 500 tokens, the collection becomes about 17,000 chunks. Each is embedded once and embedded again only when its document changes.
- A site manager asks, “What do we need in place before hot works on a flat roof?” Hybrid search returns 30 candidate chunks, already filtered to this company and to documents the site manager may see.
- A reranker picks the best five. With instructions and the question, the model receives about 3,000 input tokens and writes an answer of about 300 tokens.
- The answer lists what the company’s own hot works procedure requires, such as the permit and the fire watch, with a citation to the section each point comes from.
The cost per question follows the same sum as any model call. It is 3,000 input tokens at the input rate plus 300 output tokens at the output rate, plus a small cost for embedding the question and reranking. Our guide to what an AI feature costs per call shows how to put rates into that sum.
When RAG is the wrong tool
- Totals and calculations. “How much did we invoice in March?” is a database query. Give the model a tool that runs the query, instead of searching invoice PDFs.
- Whole-document tasks. Summarising one contract or comparing two versions needs the whole text. If it fits in the model’s context window, send it.
- Small, stable knowledge. A two-page policy can go in the prompt every time, with no index at all.
- Out-of-date or contradictory documents. RAG repeats what your documents say. If the drive holds three versions of a procedure, tidy it first or mark which version is current.
- Answers that must be exact every time. Where a wrong answer has legal or safety consequences, use RAG to find and show the source, and let a person read it.
Questions people ask
What is the difference between RAG and fine-tuning?
RAG gives a model your information at the moment it answers, by retrieving relevant passages. Fine-tuning changes the model itself by training it further on examples, which suits teaching a style or format better than adding facts. For answering from documents that change, RAG is usually the better fit, because updating a document updates the answers.
What is an embedding?
An embedding is a list of numbers that represents the meaning of a piece of text, produced by an embedding model. Texts with similar meanings have embeddings that are close together, which lets a system find relevant passages even when they use different words from the question.
Does RAG stop AI from making things up?
It reduces the problem without removing it. A model can still misread a passage or combine two passages wrongly. Citations, instructions to say when the answer is not in the sources, and a test set of questions with known answers keep the risk visible and measurable.
How do you measure whether a RAG system works?
Build a test set of real questions with the passages that answer them. Measure how often the right passage appears in the retrieved results, then how often the final answer is correct and properly cited. Run the test set again after every change to chunking, search or prompts.