On this page 11 sections
Prompting fixes instruction and format problems. Retrieval fixes knowledge problems. Fine-tuning fixes behaviour problems, and it does not reliably teach facts. Most teams asking about fine-tuning describe a retrieval problem, so the expensive lever gets pulled for something the cheap one solves better. Try them in that order, with a test that says when to stop.
“Should we fine-tune?” is almost never the real question. The real question is what kind of wrong the system currently is.
Key takeaways
- The three levers answer three different questions. Match the lever to the symptom, not to the budget.
- Fine-tuning teaches form and behaviour. Facts learned this way cannot be updated without retraining, and cannot be cited.
- Retrieval is the right answer for anything that changes, anything that needs a source, and anything with permissions attached.
- Prompting is not the amateur option. It is the fastest to change and the easiest to get wrong at length.
- Cost of ownership matters more than build cost: who is allowed to change this, and what happens when the base model is deprecated.
Who this applies to
You have a working system that is wrong in some specific way, and someone has proposed fine-tuning as the fix. Or you are scoping a build and deciding how much of the behaviour to bake in.
If you have not yet measured what kind of wrong it is, that comes first. Measuring whether an AI system works is the prerequisite, because all three levers are indistinguishable from each other when the only evidence is an impression.
The question each lever answers
| Symptom | The lever | Why |
|---|---|---|
| It does not know our products, prices, policies | Retrieval | The facts change, need a source, and are often permissioned |
| It ignores an instruction, or follows it inconsistently | Prompting | The instruction is present but competing with others |
| Output shape is unreliable | Structured output, then prompting | A schema constraint is cheaper and stricter than either alternative |
| Tone is wrong, or the format is house-specific and hard to describe | Fine-tuning | Style is easier to demonstrate with examples than to specify in words |
| A narrow classification the base model keeps getting wrong | Fine-tuning | Small, closed task with plenty of labelled examples |
| It is correct but too slow or too expensive | Fine-tune a smaller model | The one case where fine-tuning is a cost lever, not a quality lever |
The first row is the one that gets misdiagnosed. “The model does not know our returns policy” sounds like a knowledge gap in the model, so training on company documents sounds like the fix. It is a knowledge gap in the request, and the fix is putting the policy in front of the model at the moment it answers.
Why fine-tuning does not teach facts reliably
Fine-tuning adjusts the model’s weights on examples. It is very good at shifting the distribution of how the model answers, and poor at installing a specific fact you can then depend on.
Three consequences follow, and each one is a reason to prefer retrieval for knowledge:
You cannot update a weight the way you edit a document. When the returns window changes from 30 days to 14, a retrieval system needs one paragraph rewritten. A fine-tuned model needs another training run, and until it happens the system is confidently wrong.
There is no source to show. A fine-tuned answer arrives with no passage attached, so it cannot be cited, cannot be checked by the customer, and cannot be audited later. This is exactly the property that makes unsupported answers the measurable metric rather than wrong ones.
Teaching new facts can make hallucination worse. Training a model on statements it has no basis for teaches the form of confident specificity along with the content. The model learns that answering this kind of question with a precise-sounding detail is what it does here, and it generalises that behaviour to questions where it has no detail.
Parameter-efficient methods have made fine-tuning far cheaper to run. They did not change what it is good at.
What fine-tuning is genuinely for
It is not a last resort, and dismissing it is as lazy as reaching for it first. Four cases where it is the right tool:
A house format you cannot describe. If your output has to match a document style with a hundred small conventions, twenty examples teach it better than two pages of prose instructions, and the prompt gets shorter rather than longer.
Tone that survives. Prompted tone drifts as conversations lengthen and as prompts accumulate. Trained tone does not.
A narrow classification with real labelled data. Routing, triage, extraction into a fixed taxonomy. Closed task, thousands of examples, no facts required.
Cutting cost and latency. Fine-tune a small model on the outputs of a large one for your specific task. This is the one place fine-tuning routinely pays, and it is a cost decision rather than a quality one - which means it needs a before number or you will not be able to tell what you gave up.
The order, and when to stop
Work up the list. Stop at the first lever that closes the gap, and require a measurement before moving to the next.
1. Fix the prompt, then shorten it. Most under-performing prompts are not too short. They have accumulated a year of instructions, several in tension, and the model is resolving the conflict in a way you did not intend. Removing an instruction is a change worth testing, which is what an evaluation set makes safe - see regression testing prompts like code.
2. Fix retrieval. If the answer exists in your documents, the failure is usually that the right passage was never retrieved, not that the model could not use it. Measure retrieval hit rate separately from answer quality, because end-to-end scoring hides which half is broken. The six causes and the test for each are in why retrieval misses the answer that is there.
3. Constrain the output. If the complaint is shape rather than substance, structured output is stricter and cheaper than training.
4. Then consider fine-tuning, with a held-out set you have not touched, and a specific number you expect it to move.
The exit test at each step is the same: what number would tell me this worked, and do I have it from before the change? If the answer is no, the next lever will not help either, because you will not be able to tell whether it did.
What each one costs to own
Build cost is the number that gets discussed. Ownership cost is the one that decides how the system ages.
| Prompting | Retrieval | Fine-tuning | |
|---|---|---|---|
| Time to first version | Hours | Days to weeks | Days, plus data preparation |
| Cost to change | Minutes | Edit a document | A training run |
| Who can change it | Anyone with review access | Whoever owns the content | An engineer, with data |
| Facts stay current | Only if in the prompt | Yes, by editing the source | No, until retrained |
| Answer can cite a source | No | Yes | No |
| Permissions per user | No | Yes | No |
| Survives a base model change | Usually, with regression testing | Yes | No, retrain against the new base |
That last row is the one that surprises people. A fine-tune is tied to a base model version, and base versions get deprecated on a published schedule. When that happens you are not choosing whether to retrain, you are scheduling it - the same forced-migration problem described in your provider changed the model, with more work attached.
The permissions row rules fine-tuning out entirely for a class of systems. If two employees must see different answers from the same question, the difference has to live in retrieval, because a model’s weights cannot be filtered per user. That is the subject of RAG permissions.
What we actually do
We have never opened an engagement with fine-tuning, and we have said so to clients who had already budgeted for it. It has cost us work at least twice, because “you do not need the expensive thing” reads as a smaller engagement, and sometimes the buyer has already sold it internally as the plan.
The reasoning is the same every time. Fine-tuning is the only one of the three levers that is hard to reverse, hard to explain to the person who owns the content, and dependent on a base model somebody else controls. Everything it does for knowledge, retrieval does better and cheaper. What is left - format, tone, narrow classification, distilling for cost - is real, and it is a smaller share of the requests we get than the requests suggest.
Our default sequence on a support or knowledge system is retrieval first with hit rate measured separately, a deliberately short prompt, structured output where the shape matters, and fine-tuning considered only once there is a held-out set and a number it is expected to move. On roughly nine builds in ten, that number never gets a chance to justify the work, because the gap closes earlier.
When this is not worth deciding carefully
If the system handles low volume and low stakes - an internal drafting helper, a prototype in front of five people - use the best model with a good prompt and move on. The comparison in this article costs more to run than the difference is worth at that scale.
The same applies if you are still deciding what to build. Architecture decisions taken before the workflow is settled tend to be re-taken, and the cost of the analysis is the loss.
And if your volume is genuinely small, a fine-tune to save cost is arithmetic that does not work. The training run, the data preparation and the eventual forced migration are fixed costs; you need the traffic to amortise them.
Frequently asked questions
How many examples does fine-tuning need?
For style and format, low hundreds of good examples usually beats thousands of mediocre ones. For classification, more is genuinely better. The binding constraint is almost always label quality rather than count, and preparing that data is the part teams underestimate.
Can we fine-tune and use retrieval together?
Yes, and for a specialist domain it is often the right combination: fine-tune for the form of the answer, retrieve for its content. The mistake is using fine-tuning to supply content that retrieval should be supplying.
Is fine-tuning cheaper now that adapters exist?
The training run is much cheaper. The data preparation, the evaluation and the forced migration when the base model is deprecated are not, and they are the larger share of the cost over a year.
Our vendor says their fine-tuned model is more accurate. How do we check?
Ask what it was measured against, whether the cases were held out, and what the base model scored on the same set. Without those three, “more accurate” is a claim about a demo. The same test applies to any accuracy number in a proposal - see how to evaluate an AI agency proposal.
What if the answers are wrong in a way retrieval clearly cannot fix?
Then check whether the task is genuinely ambiguous before reaching for training. If two of your own experts would answer differently, no amount of fine-tuning produces a right answer, and you have found the ceiling instead - the measurement is in testing against the human baseline.
Next step
If someone has proposed fine-tuning and nobody can say which number it is expected to move, that is worth an hour before it is worth a budget. The AI consulting and strategy engagement covers this choice as a scoping question, and the answer is frequently the cheaper lever.
Related: Context windows and long documents · Structured output: reliable JSON · RAG permissions · Regression testing prompts like code · AI consulting and strategy