Retrieval-augmented generation (RAG): making AI answer from your own data

Imagine you run a school. You have a fifty-page student handbook, a code of conduct and timetables, and you want a smart assistant that answers parents’ questions accurately. If you ask a general AI tool “when does registration open at my school?”, it will not know, because your documents were never part of its training data – and it might invent a date that sounds convincing. The solution most organisations rely on today is called retrieval-augmented generation, or RAG. In this article we explain the idea from scratch, walk through its parts step by step, and show when it is the right solution and when it is not.

The problem: a clever model that does not know your documents

As we explained in what large language models are, a model learns from training data that stops at a certain date and does not include your organisation’s private documents. That creates two problems. First, the model does not know new or private information. Second, when it does not know, it may generate a convincing but untrue answer – the behaviour known as hallucination.

The core idea of retrieval-augmented generation is very simple: before asking the model to answer, we search our documents for the parts most relevant to the question, place them in front of it in the prompt, and ask it to answer from them alone. It is like an “open book” exam: the student does not need to memorise everything, only to know where to look and how to use what they find.

How does it work? Five stages

1. Chunking the documents

We cannot put a fifty-page handbook into every question, so we split it into small pieces, such as one or two paragraphs. The size of a piece is an important decision: pieces that are too small lose context, and pieces that are too large add information that does not help. A common practice is to let each piece overlap slightly with the next, so that an idea is not cut in half.

2. Embeddings

We turn each piece into a long list of numbers called a vector, using a model built for the purpose. The magical property is that texts close in meaning get vectors that are mathematically close, even when their words differ. So “When does registration start?” sits close to a paragraph saying “Admissions open in the first week of September”, although the two share almost no words.

3. Storing in a vector database

We store the vectors together with the original text and extra details such as the document name and page number. Vector databases are designed to search quickly for the “nearest neighbours” among millions of vectors.

4. Retrieval

When the user’s question arrives, we turn it into a vector in the same way and search for the pieces closest to it. Many systems combine this semantic search with traditional keyword search, because the latter is better at names and exact numbers such as a course code or a product reference.

5. Generation

We place the retrieved pieces in the prompt with clear instructions, for example: “Answer the question using only the texts below. If the answer is not there, say you do not know, and name the document you relied on.” This is where what we learned in prompt engineering matters.

The following short video from IBM Technology explains the idea clearly:

RAG or fine-tuning?

A question that comes up often: why not simply retrain the model on our documents? The table below summarises the difference:

In practice, retrieval-augmented generation is the most suitable starting point for most projects, and fine-tuning is an extra option when you need specialised behaviour.

Common mistakes when building a RAG system

• Disorganised documents: scanned PDFs without good text extraction, or tables that turn into jumbled text. A system is never better than its sources.
• Ignoring updates: if the rules change and the database is not updated, the system will answer confidently from an old version.
• Retrieving too many pieces: flooding the model with irrelevant text scatters the answer. Start with a small number of pieces and measure the result.
• Not letting the system say “I don’t know”: an honest admission is better than an invented answer.
• Forgetting privacy: the system must not retrieve one student’s private information for another. Permissions must be applied at retrieval time.

How do you measure the system’s quality?

Prepare a set of real questions whose correct answers and sources you know. Then measure two things separately: did the system retrieve the right pieces? And was the final answer correct and grounded in those pieces? Separating the two matters, because a fault may lie in the search or in the wording, and each has a different fix. Re-run the evaluation after every change to the chunk size, the embedding model or the instructions.

A worked example: an assistant that answers from the student handbook

Let us return to the school at the start of this article and follow a real question through the system. A parent writes: “Can my son change his study track after the term has started?” The question becomes a vector, and the system searches the handbook’s pieces and finds three close ones: a paragraph on “requests to change track”, one on “term dates” and one on “academic guidance”. The three paragraphs go into the prompt with the instructions, and the model writes an answer such as: “According to the student handbook, page 12, a request to change track may be submitted during the first two weeks of term after a meeting with the academic adviser.” Notice that the answer names its source, so the parent or a member of staff can check it easily. And if someone asks about something that is not in the handbook – say, the fee for a new activity that has not been announced – the right answer is: “I could not find this in the available documents; please contact the office.”

Examples of practical uses

In education: an assistant that answers students from the course material itself, pointing to the related lesson. In customer service: answers from the company’s current policies. Inside an organisation: searching meeting notes and technical documents in natural language. And when retrieval is combined with the ability to use tools, we arrive at what are called AI agents.

If you want to build a system like this yourself, you will need a good grounding in Python programming, which you can start in the academy’s courses. In the artificial intelligence course we meet these techniques and try them in practice. For further resources visit the learning library, and to order your next steps try the Find your path tool.