RAG Architecture Explained: Retrieval-Augmented Generation

A language model can draw on two things: what training left in its parameters, and the text in front of it. Your company's leave policy, last week's release notes and the contract signed yesterday are in neither. Ask about them and a model will either say it does not know or, worse, write something plausible.

RAG architecture solves this without retraining anything. Before the model answers, a search step finds the relevant passages and puts them into the prompt. The model then answers from text it can actually see.

Why it exists

The 2020 paper that named the technique set out the problem clearly. Language models store factual knowledge in their parameters, but their ability to access and manipulate it precisely is limited, and showing where an answer came from and updating what the model knows were unsolved. The proposal was to pair the model with a second, external memory that can be searched (Lewis et al.: Retrieval-Augmented Generation).

That gives three practical benefits:

• Current and private knowledge. The documents can be yours, and can change daily.
• Evidence. The answer can point to the passage it used.
• No training. Updating knowledge means updating documents.

Anthropic's glossary describes it as grounding the model's response in evidence that is retrieved at runtime and passed into the context window (Anthropic: glossary).

The four stages

The name is the last three stages in order: retrieval, augmented, generation. See embeddings explained for the vectors and vector databases explained for where they are kept at scale.

A pipeline you can run

The script is a complete RAG system in about forty lines. The knowledge base is six Python Enhancement Proposals, public-domain documents from peps.python.org, split into 148 sections. Embeddings come from a small open model (all-MiniLM-L6-v2), and answers from a 3-billion-parameter open model served on the same laptop by Ollama (Ollama: API). Nothing leaves the machine.

It asks three questions, chosen to show three different outcomes.

import json
import urllib.request

import numpy as np
from sentence_transformers import SentenceTransformer

# 1. INDEX (done once): split the documents, embed every piece, keep the vectors.
sections = json.load(open("../../genai/pep_sections.json", encoding="utf-8"))
embedder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
vectors = embedder.encode([s["title"] + "\n" + s["text"] for s in sections], normalize_embeddings=True)

PROMPT = """Answer the question using only the context below. Quote the rule in one or two sentences
and name the section it came from. If the context does not contain the answer, reply exactly:
I cannot find that in the documents.

Context:
{context}

Question: {question}"""


def generate(prompt):
    body = {"model": "llama3.2:3b", "prompt": prompt, "stream": False,
            "options": {"temperature": 0, "seed": 42, "num_predict": 90, "num_ctx": 4096}}
    request = urllib.request.Request("http://localhost:11434/api/generate", data=json.dumps(body).encode(),
                                     headers={"Content-Type": "application/json"})
    return json.load(urllib.request.urlopen(request, timeout=1800))["response"].strip()


def answer(question, k=3):
    # 2. RETRIEVE: embed the question and take the k nearest pieces.
    scores = vectors @ embedder.encode(question, normalize_embeddings=True)
    top = np.argsort(-scores)[:k]
    # 3. AUGMENT: put those pieces into the prompt, cut to 250 words each.
    context = "\n\n".join(f"[{sections[i]['id']}: {sections[i]['title']}]\n" + " ".join(sections[i]["text"].split()[:250])
                          for i in top)
    # 4. GENERATE: the model answers from the prompt.
    reply = generate(PROMPT.format(context=context, question=question))
    print("Q:", question)
    print("   retrieved:", ", ".join(f"{sections[i]['id']} {sections[i]['title']!r} ({scores[i]:.2f})" for i in top))
    print("   answer:", " ".join(reply.split()))
    print()


answer("What is the maximum line length in Python code?")
answer("How should I compare a value with None?")
answer("Who won the 2023 Cricket World Cup?")

Q: What is the maximum line length in Python code?
   retrieved: pep8-5 'Maximum Line Length' (0.72), pep8-1 'Introduction' (0.45), pep8-3 'Indentation' (0.44)
   answer: The maximum line length in Python code should be limited to 79 characters, as stated in PEP8-5. Section: [pep8-5: Maximum Line Length]

Q: How should I compare a value with None?
   retrieved: pep572-27 'Why not just turn existing assignment into an expression?' (0.34), pep572-22 'Special-casing conditional statements' (0.31), pep572-24 'Lowering operator precedence' (0.28)
   answer: There is no information in the context about how to compare a value with None. I couldn't find that in the documents.

Q: Who won the 2023 Cricket World Cup?
   retrieved: pep20-1 'Abstract' (0.11), pep572-17 'Capturing condition values' (0.09), pep8-15 'Comments' (0.09)
   answer: There is no information about the 2023 Cricket World Cup in the provided context. Context does not contain the answer.

Question 1: it works

The section "Maximum Line Length" came back first with a similarity of 0.72, well ahead of the rest. The model answered 79 characters and named the section. This is the system doing what it was built for: the answer is in the documents, the right passage was retrieved, and the model used it.

Question 2: retrieval fails, and the model says so

The style guide does say how to compare with None. It is in a section called "Programming Recommendations". That section was not retrieved. Three sections from a different document came back instead, with weak scores around 0.3, and none of them contains the answer.

The model's reply is the correct one in the circumstances: it could not find the answer in what it was given. It did not fall back on what it remembers about Python.

Two lessons sit in this result. First, most RAG failures are retrieval failures. The generator can only be as good as the passages it receives. Second, the cause here is how the documents were split. "Programming Recommendations" is a single section of more than 1,300 words covering dozens of rules. Its one embedding has to stand for all of them, and the embedding model reads only the first 256 word pieces of any input. The rule about None is far below that cut-off. Splitting that section into smaller pieces fixes this question. See chunking strategies for RAG.

Question 3: outside the documents

Nothing in six documents about Python concerns cricket. The best similarity was 0.11, and the model declined to answer.

Look at the wording, though. The prompt asked for one exact sentence when the answer is missing. The model wrote its own versions instead. A small model follows format instructions loosely, and code that checks for the exact string would not recognise either refusal. If your application needs to detect "no answer", test how your model really phrases it, or ask for a structured output.

When you do not need it

RAG is the answer to a knowledge base that does not fit in the prompt. If it fits, the simplest approach is to include all of it. Anthropic's guidance puts the line at about 200,000 tokens, roughly 500 pages, below which the whole knowledge base can go into the prompt (Anthropic: contextual retrieval).

Reach for RAG when the material is too large for that, changes often, or must be restricted so that different users see different documents.

Making it better

The pipeline above is the minimum. Production systems add to each stage:

• Better splitting. The fix for question 2.
• Hybrid retrieval. Embeddings miss exact terms such as codes and names. Combining them with keyword search and merging the rankings covers both. See semantic search vs keyword search.
• Context for each piece. A chunk that says "revenue grew by 3%" does not say whose revenue. Anthropic reports that adding a short description of where each chunk came from reduced failed retrievals by 49%, and by 67% when combined with reranking.
• Reranking. Retrieve twenty candidates cheaply, then use a stronger model to pick the best few.
• Measurement. A set of questions with known answers, run after every change. See retrieval evaluation.

For a full application built on this pattern, see building a document Q&A chatbot with RAG. For the alternative of changing the model itself, see fine-tuning vs RAG.

Common mistakes

• Tuning the prompt when retrieval is the problem. Print what was retrieved before changing anything else.
• Indexing whole documents or very long sections.
• No instruction for the case where the answer is missing, so the model fills the gap from memory.
• Trusting similarity scores as confidence. 0.34 was a miss here, but there is no universal threshold.
• No test questions. Without them, every change is a guess.
• Ignoring access control. If a user may not read a document, it must not be retrievable for them.

Retrieval-augmented generation and document Q&A systems are built step by step in the Generative AI course.

Sources

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv) — Lewis et al.
• Contextual Retrieval in AI Systems (Anthropic) — Anthropic
• Glossary (Claude Platform Docs) — Anthropic
• all-MiniLM-L6-v2 model card (Hugging Face) — Sentence Transformers
• API introduction (Ollama documentation) — Ollama
• Python Enhancement Proposals (Python Software Foundation) — Python Software Foundation