RAG, Fine-Tuning or Prompt? Right Knowledge for Your AI

Antonio Blago
Antonio Blago
Add as preferred source on Google
3 readers
Agent mode (Markdown)
Share
AI summary
  • Without shop data, the language model guessed the Tempo 2 runs larger for sprinters, a completely made-up answer that sounded convincing.
  • With RAG, the same model correctly answered the Tempo 2 runs half a size small, citing the sizing page as source from three retrieved documents.
  • RAG is the right route when company-specific knowledge is large, changes frequently, requires sources, or needs permission levels per user.
  • The recommended order is first prompting, then RAG, then fine-tuning, with numbers and live data fetched through SQL or API instead.

Generated with AI, the details are in the article.

Table of contents

You want to order running shoes online and first ask the shop chatbot whether the shoe runs large or small. The answer: rather large. So you order one size down, the parcel arrives, and the shoe pinches. At that moment the chatbot guessed. The shop never gave it the information it would have needed: the exact measurements, the fit and the feedback from other customers.

This article belongs to my YouTube series "Daten für KI" (Data for AI) and is the written version of episode 1. The video is in German. The article answers the question that comes before every AI project: how does your company's knowledge get into a language model, by prompt, by RAG or by fine-tuning? At the end you get six criteria and four yes-or-no questions to decide it for your own case.

Episode 1 of the series "Daten für KI" on YouTube, in German: watch the video on YouTube.

The test: a chatbot guesses, and sounds sure while doing it

For the video I made up a fictional running shoe shop. It is called Sprintwerk, the shoe is called Tempo 2. Six documents go with it: product page, price list, sizing and fit with inner lengths, returns, FAQ and customer reviews. The language model is qwen2.5:3b, running locally through Ollama on my machine.

The answer without shop data

I ask the model whether the Tempo 2 from Sprintwerk runs large or small. The shop does not exist, so the model cannot know it. An answer arrives immediately anyway: the shoe runs larger and is built especially for sprinters. It sounds convincing and is completely made up. The answer stands here unchanged.

The same question with RAG

Then I ask the same model the same question again, this time with RAG in Open WebUI. Before the model answers, the system searches the documents of the collection "Sprintwerk Shop" and hands it the matching passages. Now the answer reads: the Tempo 2 runs half a size small, with the advice to order half a size up. As its source the system names the file groessen_passform.md (the sizing and fit page), and three sources were retrieved. The screenshots from the demo show the German interface, so they appear only in the German version of this article.

The model is the same in both cases. The difference lies entirely in the data it gets to see. With the first answer you would have ordered the wrong size, with the second the right one.

Why language models hallucinate

A model predicts the next word

A large language model is trained on huge amounts of text and does one thing really well: it predicts which word comes next. What it learned along the way sits in billions of numbers, the weights. It looks nothing up, it formulates what sounds plausible. Whether that is true depends on the data it gets to see.

Two gaps every model has

That leaves two gaps. What happened after the training cutoff is unknown to the model. And your internal documents were never part of the training: the fit of your shoe, your return period, your current prices. If you ask anyway, you often get an answer that sounds plausible and is wrong. That is called a hallucination.

In a kickoff with an industrial client in October 2026 one more point came up that I now mention in every project: language models never reach one hundred percent accuracy. Anyone building an automation therefore needs a check outside the model. In our case that is a script with a validator that feeds errors back to the model until the output is correct. For company knowledge this means it has to get into the model with every question somehow, and the real question is only by which route.

Visibly AI: your SEO copilot

Projects, rankings, Search Console data and a copilot that runs the analysis for you. Start for free.

Open Visibly App

Happy clients and case studies: All case studies

Three ways to the right knowledge: prompt, RAG, fine-tuning

There are three ways. They do not exclude each other, and most systems combine prompting and RAG.

WayHow knowledge reaches the modelStrengthLimit
Prompt engineeringKnowledge and rules sit directly in the promptWorks at once, no extra technologyLimited text length, every question gets more expensive
RAGThe system searches matching passages and passes them alongCurrent, with sources, model stays unchangedTexts only, quality depends on the data
Fine-tuningThe model is retrained on your own examplesStyle, format and terminology stickLearns facts unreliably, retrain when data changes
The three ways compared.

Prompt engineering: the first step

In prompt engineering you write knowledge and rules directly into the prompt. Your return policy in three sentences fits in easily. It works at once. The limit is the amount of text that fits into a prompt, and every question gets more expensive with the full text attached. For me prompting is always the first step, because it costs little and quickly shows what is still missing.

RAG: search, add, answer

RAG stands for retrieval-augmented generation. Retrieval means the system picks out the matching passages of text. Augmented means those passages go into the prompt. In the generation step the model should answer only from them and name the source. The model itself stays unchanged. In the shoe example there are six documents. With two thousand product pages that no longer fits into a prompt, and that is exactly when RAG pulls out only the passages that match the question.

Fine-tuning: mostly changes the how

In fine-tuning you retrain the model on your own examples. That mainly changes style, format and terminology, for example so that it answers in your shop's tone. That the Tempo 2 runs half a size small is something it learns rather unreliably, and with the next price list you would have to train again. Fine-tuning changes how the model answers. If you have knowledge that changes and that you have to prove, RAG is the standard route.

When you need RAG

I look at six criteria:

  1. It is about company-specific knowledge.
  2. The material no longer fits into a prompt.
  3. The content changes.
  4. Sources are mandatory.
  5. Not every user may see everything.
  6. Later it should be traceable where an answer came from, for an audit for example.

If two or more apply, RAG is usually the right route. I take the permissions point especially seriously. In a first call in October 2026 a company described exactly this starting point: the chatbot gives generic answers, the knowledge is spread across Excel lists, the merchandise system and the ticket system, and a pilot is meant to start with permission levels. There the outcome also depends on who may see which documents at all.

When you do not need RAG

The other side matters just as much. RAG is strong on texts, and other kinds of data have better tools:

  • A single document goes straight into the prompt.
  • Revenue and key figures you query from the database with SQL.
  • Stock levels, for example whether the Tempo 2 is available in size 44, you fetch live through an API or via MCP.
  • Calculating and summing is done by normal code.
  • Style and tone you get at the start with a few examples in the prompt.

I see this with numbers all the time. A guest on my podcast told me that the same SEO interface, in the same chat and for the same keyword, once returned a search volume of roughly twenty thousand and once of roughly two hundred. Numbers therefore belong in a query with a fixed result, and the model only words that result.

I build it the same way in Visibly AI, my own SEO platform: numbers live in a classic database, texts in a vector database for RAG. For the shoe shop it would be the same. The fit is in the product text, the stock count in the inventory, and each kind of data gets its own tool. To see how I store knowledge for AI myself, have a look at my open-source project SkillMind, a memory for AI coding assistants that also works with a vector database (skill-mind.com).

Six typical cases and four questions for your case

Six cases at a glance

CaseFitting route
Service bot for product manualsRAG
"How much revenue did we make in October?"SQL
Summarizing a contractWhole text into the prompt
Answering emails in the brand voiceFirst the prompt, maybe fine-tuning later
"Is article 4711 in stock?"API
Internal policies with access rightsRAG with permission filter
Six typical cases and the route I pick for each.

Four yes-or-no questions

  1. Does your knowledge change more often than once a month?
  2. Do the answers need a source?
  3. Do you need permissions per user?
  4. Do you have 50 to 100 reviewed examples?

The more often you say yes to the first three, the more likely RAG is the right route. The fourth question belongs to fine-tuning: only with 50 to 100 reviewed examples is it worth even looking at it. To run this for your own case, take five data sources and go through the four questions once. Not every source needs RAG, and you see that at a glance.

If you would rather not set this up alone: in my workshops on AI automation for SEO (in German) we work on data sources from your daily work, the skill shop has skill sets and courses, and when data protection sets the questions, the course Setting up Claude in a privacy-compliant way helps.

Conclusion

My order is clear: first prompting, then RAG, then fine-tuning. Prompting costs the least and shows you what is missing. RAG joins in as soon as your knowledge is large, changes or has to be backed up. Fine-tuning is the last step, when style and format should stick and you have reviewed examples. Numbers and live data you fetch through SQL or an API. The next episode is about what many AI projects hang on, the data itself: on 62 support tickets I show how you see in ten minutes what is going wrong in your stock of data. More help with setup is in the skill sets.

Sources: own conversations with clients and prospects and own podcast, October 2026, transcripts local, anonymized. Demo: fictional shop "Sprintwerk" with sample data, local model qwen2.5:3b via Ollama, RAG in Open WebUI.

Free SEO tools

Keyword ranking, SSR text check, sitemap audit and more, right in the browser and without signup.

See SEO tools

Happy clients and case studies: All case studies