Fine-Tuning vs RAG: which approach fits when? Status May 2026

What is fine-tuning, what is RAG?

Fine-tuning and RAG (retrieval-augmented generation) are the two main ways to adapt a language model to a specific domain. They solve different problems and are often combined in practice, not chosen as alternatives.

Fine-tuning changes the model's weights. From a general model you get a specialised one that writes in a certain style, follows a certain format, or understands a certain vocabulary. Classical full fine-tuning (retraining all weights) is very expensive – for a 70B-parameter model, several thousand GPU-hours, easily in five figures. As of May 2026 the standard is PEFT (parameter-efficient fine-tuning), especially LoRA (low-rank adaptation) and QLoRA. PEFT trains only a small adapter matrix (typically 0.1-1% of model weights); the original model stays untouched. Cost: CHF 50-500 per run instead of CHF 5000-50000.

RAG leaves the model untouched and at answer time supplies it with relevant text passages from your own document library. The model answers based on those passages and can cite them. For detail see the sister page retrieval-augmented-generation.

The core difference: fine-tuning teaches the model a new BEHAVIOUR (writing style, format, classification), RAG gives the model new FACTS (knowledge not in training or that changes). Each solves different problems. They can also be combined: a fine-tuned model that gets up-to-date sources via RAG.

Why this matters now

In May 2026 "fine-tune or RAG?" is no longer a principled question but a tactical one. Three developments have reshaped the picture.

First: PEFT/LoRA makes fine-tuning affordable. Until 2023, fine-tuning large models was a corporate-only affair. As of May 2026, with QLoRA and 4-bit quantisation (see was-ist-quantisierung), you train a Llama-3.1-70B adapter on a workstation with 2x RTX 4090 in 12-48 hours, for CHF 50-200 in power plus hardware depreciation. Hugging Face PEFT provides the software, OpenAI offers managed fine-tuning API, Anthropic a program access, Google Vertex AI fine-tuning. The barrier is low – but being able does not always mean being worth it.

Second: RAG is more robust in compliance contexts. A fine-tuned answer carries learned knowledge INSIDE the model – it cannot cite or prove where the information comes from. For revFADP DPIA, EU AI Act Art. 12 logging and professional secrecy (SCC 321) this is a drawback. RAG, by contrast, ships the source with every answer. In fiduciary, legal and insurance, RAG therefore continues to dominate in May 2026.

Third: context windows are so large that small knowledge bases fit in the prompt. the current top Claude model has 1 m tokens, Gemini 2 has 2 m, GPT-4.1 has 1 m. A 200-page guideline (about 100k tokens) fits entirely in one call – faster and simpler than RAG. Only from several thousand pages onward does RAG pay off. Fine-tuning in this world is used less as "knowledge injection" and more as "format and style adaptation".

The result: in May 2026 the majority answer for SMEs is "RAG, not fine-tuning". Fine-tuning is a specialised tool for recurring format tasks (e.g. consistent reminder-letter structures), not the default for knowledge applications.

How both differ in practice

We compare along six dimensions.

Data freshness. RAG is always current – new documents are indexed and immediately retrievable. Fine-tuning is static – new knowledge demands a new training run. For an SME whose knowledge changes monthly (tax circulars, association guidelines), RAG is the default.

Provability of the answer. RAG shows citation and source. Fine-tuning cannot. For applications with revFADP DPIA, EU AI Act Art. 26 logging, Art. 957a CO bookkeeping audit or professional secrecy, RAG is mandatory; fine-tuning is no substitute.

Behaviour control. Fine-tuning can enforce consistent behaviour (e.g. "always answer in formal Swiss-German Du/Sie without Anglicisms", "always write reminder letters in the same structure pattern"). RAG can do this only via a lengthy system prompt – which eats token budget. With very rigid format requirements, fine-tuning is more efficient.

Capex vs opex. Fine-tuning is capex: one-time CHF 50-500 for PEFT, then only inference costs. RAG is opex: no upfront, but per query embedding + retrieval + model call (~CHF 0.002-0.02). At very high query volumes, fine-tuning becomes attractive arithmetically; at low volumes, RAG is cheaper.

Operational complexity. Fine-tuning is "done" after one-time training and needs no additional infrastructure. RAG demands a vector DB, embedding pipeline, chunking strategy – several moving parts that can each break.

Data protection. With fine-tuning, training data leaves your domain (with managed services). With RAG, the original documents stay in your own vector DB – only the relevant passage goes to the model. RAG is the more revFADP-friendly variant, especially with a local Qdrant instance.

Hybrid in practice. May 2026 we often see: a PEFT adapter for style and format on a medium-sized open-weight model (Llama-3.1-8B, Mistral-7B, Phi-3-medium), combined with RAG for facts. That gives consistent behaviour plus current facts plus source attribution, at moderate inference cost.

Decision workflow in 6 steps

1. 01Classify task type: knowledge application (Q&A, research) or behaviour application (format, classification, style)?
2. 02Check compliance: must the answer be citable, is audit logging mandatory? If yes, RAG is mandatory; fine-tuning is no substitute.
3. 03Estimate data volume: less than 100k tokens persistent? Then put it in the prompt, no RAG, no fine-tuning. More? Then RAG.
4. 04Check format consistency: must every call produce the same rigid format? If yes, consider fine-tuning as a complement.
5. 05Build a prototype with few-shot prompting: in 1-3 days check whether the task is solvable and whether RAG/fine-tuning is justified.
6. 06For production demand: build RAG first (or prompt-only with large context); only after months, if a clear optimisation potential is visible, add fine-tuning.

When fine-tune, when RAG, when hybrid

RAG is the right choice when: (a) the answer lives in documents that change, (b) provability or source citation is needed, (c) compliance logging is mandatory (fiduciary, lawyer, insurance), (d) different clients have different data (client separation via filter), (e) the model itself is good enough and you do not want a behaviour change.

Fine-tuning is the right choice when: (a) you need consistent format/style that looks the same every call, (b) one very narrow specialised task (e.g. classifying receipt type among 12 categories) that you run a million times, (c) you want a small model optimised to your task that runs faster than a large general one, (d) data sensitivity forces an own model (local Llama with own adapters).

Hybrid is the right choice when: (a) you need consistent writing format AND current facts, (b) industry-specific language AND provable sources, (c) an own local model AND growing document knowledge. Example: a law firm system fine-tuned on Llama-3.1-8B with Swiss legal vocabulary, which consults current case law and concrete client files via RAG.

Concrete SME practice May 2026: An 8-person fiduciary with a client FAQ chatbot starts with RAG on a cloud model (Claude or GPT-4) plus Qdrant. Takes 1-2 weeks, costs CHF 3000-8000 setup plus CHF 50-200/month running. Fine-tuning only becomes a topic when, after 6-12 months, a clear format pattern emerges that pays off to automate, OR when data-protection reasons force migration to an own model.

When NEITHER of the two

Three constellations where you need neither fine-tuning nor RAG.

First: the task is general world knowledge or generic language processing. "Write me an email reply to this inquiry", "Summarise this text", "Fix this table". The bare model is enough. Fine-tuning would be overkill, RAG would be pointless without a knowledge source.

Second: the data volume is small. A 30-page guideline (15k tokens) fits entirely in Claude/GPT/Gemini context. Paste the full text into the prompt – faster, simpler, more deterministic than RAG, cheaper than fine-tuning. Only with a persistent knowledge base from a few hundred thousand tokens onward does RAG pay off; only with a clearly recurring format pattern does fine-tuning pay off.

Third: fast prototyping. Whoever does not yet know whether the application is even sensible builds NO RAG system and NO fine-tuning. Instead: prompt engineering with examples (few-shot learning) in the system prompt. Up and running in hours, gives insight whether the idea holds. If yes, then RAG or fine-tuning as a second stage.

May 2026 pitfall: fine-tuning is often sold as a "magic tool" that fixes hallucinations. That is wrong. Fine-tuning on your own data can even induce more specific hallucinations – the model learns the format of your data and then confidently produces wrong answers in that format. RAG is the more robust hallucination control (see halluzinationen-begrenzen).

Trade-offs

FAQ

Related topics

Sources

1. Lewis et al. – Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Meta AI) · 2020-05
2. Hugging Face PEFT – Parameter-Efficient Fine-Tuning Documentation · 2026-05
3. OpenAI – Fine-Tuning Guide and Pricing · 2026-04
4. Hu et al. – LoRA: Low-Rank Adaptation of Large Language Models · 2021-06