Fine-Tuning vs RAG: Which Should You Choose?

Teams adapting a language model to their business are usually told to pick one: fine-tuning vs RAG. The framing hides the real question. The two do different jobs.

RAG changes what the model can see when it answers. Fine-tuning changes the model itself, and so how it behaves. Most projects need the first long before they need the second, and some need both.

Two things a model can lack

Anthropic's glossary draws the same line. Fine-tuning is further training that makes a model reproduce the patterns of a dataset, useful for a domain, a task or a writing style. RAG passes retrieved information into the context at run time (Anthropic: glossary).

The knowledge problem, measured

The script asks a local model eight questions about a staff handbook for a company that does not exist, so nothing about it can be in any model's training data. Each question is asked twice: alone, and with the handbook placed in the prompt.

import json
import re
import urllib.request

HANDBOOK = """Meridian Analytics staff handbook, 2026 edition.
Annual leave is 23 working days. Unused leave carries over for a maximum of 5 days.
The probation period for new staff is 4 months. The notice period after probation is 45 days.
Staff may work from home on up to 2 days each week. Core hours are 10:30 to 16:00.
Laptops are replaced every 3 years. Each employee has a training budget of 12 hours per quarter.
Expense claims must be submitted within 21 days of the expense."""

QUESTIONS = [("How many days of annual leave do staff at Meridian Analytics get?", "23"),
             ("How many unused leave days can be carried over at Meridian Analytics?", "5"),
             ("How long is the probation period at Meridian Analytics, in months?", "4"),
             ("What is the notice period after probation at Meridian Analytics, in days?", "45"),
             ("How many days a week can Meridian Analytics staff work from home?", "2"),
             ("How often are laptops replaced at Meridian Analytics, in years?", "3"),
             ("How many hours of training budget does a Meridian Analytics employee have per quarter?", "12"),
             ("Within how many days must expense claims be submitted at Meridian Analytics?", "21")]


def ask(prompt):
    body = {"model": "llama3.2:3b", "prompt": prompt, "stream": False,
            "options": {"temperature": 0, "seed": 42, "num_predict": 40}}
    request = urllib.request.Request("http://localhost:11434/api/generate", data=json.dumps(body).encode(),
                                     headers={"Content-Type": "application/json"})
    return " ".join(json.load(urllib.request.urlopen(request, timeout=600))["response"].split())


def gives(text, number):
    return re.search(rf"\b{number}\b", text) is not None


closed_right = closed_wrong_number = open_right = 0
print("answer  closed book, the question alone                       with the handbook in the prompt")
for question, answer in QUESTIONS:
    closed = ask(question + " Answer in one short sentence.")
    opened = ask(f"{HANDBOOK}\n\nUsing the handbook above, answer in one short sentence. {question}")
    closed_right += gives(closed, answer)
    closed_wrong_number += bool(re.search(r"\d", closed)) and not gives(closed, answer)
    open_right += gives(opened, answer)
    print(f"{answer:>6}  {closed[:56]:56}  {opened[:44]}")

print()
print(f"closed book: {closed_right} of {len(QUESTIONS)} correct, {closed_wrong_number} gave a wrong number")
print(f"with the handbook: {open_right} of {len(QUESTIONS)} correct")

answer  closed book, the question alone                       with the handbook in the prompt
    23  I couldn't find any information on the number of annual   Staff at Meridian Analytics receive 23 worki
     5  I don't have information on the specific policies of Mer  At Meridian Analytics, up to 5 unused annual
     4  I couldn't find any information on a company called "Mer  The probation period for new staff at Meridi
    45  I don't have access to that information.                  The notice period after probation at Meridia
     2  I don't have information about Meridian Analytics' speci  Meridian Analytics staff can work from home 
     3  I don't have information about Meridian Analytics or the  Laptops are replaced every 3 years at Meridi
    12  I don't have information about the specific training bud  A Meridian Analytics employee has 12 hours o
    21  I don't have the specific information on the time frame   Expense claims must be submitted within 21 d

closed book: 0 of 8 correct, 0 gave a wrong number
with the handbook: 8 of 8 correct

• With the question alone, the model got none right. It could not have. To its credit it said so each time and invented nothing.
• With the handbook in the prompt, it got all eight right. Same model, same parameters, no training.
• The whole fix was 85 words of context.

This is the case for retrieval in miniature. The handbook here is short enough to paste in whole. A real knowledge base is not, so a search step chooses which parts to paste: that is RAG. See RAG architecture explained. If your material is small, you may not need even that. Anthropic's guidance is that a knowledge base under about 200,000 tokens can simply go into the prompt (Anthropic: contextual retrieval).

Why not train the facts in

It seems natural to fine-tune the model on the handbook so that it "knows" it. Research has compared the two directly. A study across several knowledge-heavy tasks found that RAG consistently outperformed fine-tuning on raw documents, for knowledge the model had seen and for knowledge that was entirely new, and that models struggled to learn new facts that way (Ovadia et al.: Fine-Tuning or Retrieval?).

There are practical reasons too:

• Facts change. Leave policy changes in January. A document is edited in a minute, and a fine-tuned model has to be trained again.
• No sources. A fine-tuned model states things without being able to point to where they came from. The original RAG paper named provenance as one of the problems retrieval addresses (Lewis et al.: RAG).
• No access control. Once a fact is in the weights, every user of the model can get at it.

What fine-tuning is for

Fine-tuning earns its place when the problem is behaviour. OpenAI's documentation lists what it adds beyond prompting: more examples than fit in a context window, shorter prompts and so lower cost at scale, and a smaller, cheaper model made good at one task. It names classification, nuanced translation and content in a specific format as good fits (OpenAI: supervised fine-tuning).

Typical cases:

• A strict output format that a prompted model follows most of the time, when you need every time.
• A house style or tone across thousands of generated texts.
• A narrow, high-volume task, such as classifying tickets, where a small tuned model replaces a large general one.
• A long prompt full of instructions and examples that you pay for on every call.

The data requirement is smaller than people expect. OpenAI gives a minimum of 10 examples and reports improvements from 50 to 100.

Where hosted fine-tuning stands

A fact that changes the calculation: as read on 2 October 2026, OpenAI's documentation says the company is winding down its fine-tuning platform, that it is no longer open to new users, and that existing fine-tuned models stay available until their base models are deprecated (OpenAI: model optimization). Anthropic's glossary says the Claude API does not currently offer fine-tuning.

So for many teams, fine-tuning now means open models, trained with efficient methods on your own or rented hardware. See LoRA and QLoRA. It also means owning what comes after: evaluation, hosting, and retraining when a better base model appears. That is a larger commitment than adding a retrieval step, and a reason to exhaust the cheaper options first.

Costs side by side

An order to try things in

OpenAI's own optimization guide describes a loop of evaluation, prompting and, for some use cases, fine-tuning. In practice that becomes:

1. Write a test set. Without one you cannot tell whether anything helped. See retrieval evaluation.
2. Improve the prompt. Clear instructions and a few examples fix more than expected. See prompt engineering techniques.
3. Add the knowledge. In the prompt if it is small, by retrieval if it is not.
4. Use structured outputs if the problem is format.
5. Fine-tune if behaviour is still inconsistent, or if a long prompt is costing too much at volume.
6. Combine them when you need both: a tuned model for the behaviour, retrieval for the facts.

Common mistakes

• Fine-tuning to add facts.
• Fine-tuning before writing a single evaluation.
• Building RAG for 20 pages of text that would fit in the prompt.
• Blaming the model for missing knowledge it was never shown.
• Forgetting that a fine-tuned model is frozen on the day it was trained, and tied to a base model that will be retired.
• Treating the choice as either-or.

Fine-tuning versus prompting and retrieval-augmented generation are compared in the Advanced LLM Concepts module of the Generative AI course.

Sources

• Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs (arXiv) — Ovadia et al.
• Model optimization (OpenAI API documentation) — OpenAI
• Supervised fine-tuning (OpenAI API documentation) — OpenAI
• Glossary (Claude Platform Docs) — Anthropic
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv) — Lewis et al.
• Contextual Retrieval in AI Systems (Anthropic) — Anthropic