Short answer
Use retrieval-augmented generation (RAG) when the model needs your current, changing facts, such as policies, product data or tickets, with citations. Use fine-tuning when you need a consistent behavior, format or tone that prompting cannot achieve. Most production systems start with RAG; Zeta builds both and tests which one your use case needs.
What is retrieval-augmented generation (RAG)?
RAG gives a model the right information at the moment it answers. Your documents are split into passages, indexed for search, and the most relevant passages are retrieved and placed in the prompt alongside the question. The model then answers from that material and can cite it. The approach was formalised by Lewis et al. (2020), who showed that combining a retriever with a generator improves performance on knowledge-intensive tasks.
Because knowledge lives in the index rather than in the model, updating it is as simple as re-indexing a changed document. Access control can be enforced at retrieval time, so people only receive answers built from documents they are allowed to see.
What is fine-tuning?
Fine-tuning continues training a model on your own examples so that it adopts a behavior: a house style, a strict output format, a classification scheme or domain-specific phrasing. Parameter-efficient methods such as LoRA make this far cheaper than full retraining; the original paper reports reducing the number of trainable parameters by 10,000 times compared with full fine-tuning of GPT-3 175B.
Fine-tuning is a poor way to teach facts that change. Updating knowledge means preparing new data and training again, and a fine-tuned model cannot point to the document an answer came from.
What are the differences between RAG and fine-tuning?
| RAG | Fine-tuning | |
|---|---|---|
| Primary purpose | Supply current, specific knowledge | Shape consistent behavior, format or tone |
| Keeping knowledge fresh | Re-index changed documents | Prepare new data and retrain |
| Citations and traceability | Yes, to the retrieved passages | No built-in source attribution |
| Access control | Enforced per user at retrieval time | Anything in training data can surface |
| Upfront effort | Data pipeline, chunking, search tuning | Curated, labelled training examples |
| Running cost driver | Longer prompts and search infrastructure | Training runs and model hosting |
| Typical failure | Retrieves the wrong or incomplete passage | Confident answers that drift from the facts |
When should you choose RAG?
- Answers must reflect documents that change: policies, catalogues, contracts, tickets.
- Users need to see and verify the source of each answer.
- Different people are allowed to see different documents.
- You want to start quickly and improve iteratively without retraining.
When should you choose fine-tuning?
- Output must follow a strict structure every time, and prompting alone is not consistent enough.
- You need a specific tone or domain vocabulary across very high volumes.
- A smaller, cheaper model must match a larger one on a narrow, well-defined task.
- You have hundreds or thousands of good examples of the behavior you want.
Can you combine RAG and fine-tuning?
Yes. A common pattern is a model fine-tuned for format and tone that still receives its facts through retrieval. Start with RAG and strong prompts; fine-tune only when evaluation shows a behavior gap that prompting and better retrieval cannot close.
How do you evaluate either approach?
Build a test set before you build the system: real questions, the correct answers and the documents that support them. Then measure retrieval and generation separately, so you know which part to fix.
| Measure | Question it answers |
|---|---|
| Retrieval recall | Did the right passage appear in the retrieved set? |
| Answer faithfulness | Is every claim supported by the retrieved passages? |
| Answer correctness | Does the answer match the agreed reference answer? |
| Refusal quality | Does the system say "I don't know" when the source is missing? |
| Latency and cost | Is it fast and affordable enough at expected volume? |
What are the common failure modes?
- Poor chunking that splits a rule from its exceptions, so the model answers from half a policy.
- Stale indexes because nobody owns re-indexing when documents change.
- Leaking restricted content when permissions are not applied at retrieval — listed by OWASP as sensitive information disclosure.
- Fine-tuning on facts, which bakes in knowledge that is out of date by the next release.
- No evaluation set, so nobody can tell whether a change made things better or worse.
Sources
About the author
Zeta Consulting Engineering Team
Written and reviewed by the senior engineers who design, build and operate Zeta's AI agent, retrieval and integration projects for client platforms.