RAG, Fine-Tuning or Prompting: Choosing for Production

"Should we fine-tune or use RAG?" is the most common architecture question we get asked about AI features, and it is almost always asked too early. It is a question about implementation, arriving before anyone has settled what the model is failing at.

The three options are not competing solutions to one problem. They fix different problems, and using the wrong one produces a system that is expensive to run and no better than the version you started with.

What each one does

Prompting changes the instructions

You give the model better direction: clearer task description, worked examples of good output, an explicit format, constraints on what to do when it is unsure. Nothing about the model changes; you are getting more out of what it already knows.

Cost: essentially nothing beyond the extra tokens. Time to change: minutes.

Retrieval changes what the model can see

Retrieval-augmented generation fetches relevant material at request time and puts it in front of the model before it answers. Your product documentation, the customer's own records, last week's pricing: anything the model could not have known and must not guess at.

Cost: a storage and search layer to build and keep current, plus the tokens for whatever you retrieve on every request. Time to change: days to weeks.

Fine-tuning changes the model's defaults

You continue training a base model on your own examples so it internalises a behaviour: a house style, a rigid output format, a classification scheme specific to your domain. It shifts how the model responds by default, so you stop having to explain that behaviour in every prompt.

Cost: a curated training set (the real expense, usually hundreds to thousands of reviewed examples), the training run, and a commitment to redo it whenever the requirement moves. Time to change: weeks.

The distinction that decides it

Almost every case resolves once you name what is missing:

• The model does not know something → retrieval. Facts, documents, current data, anything private to your business or newer than the model. Fine-tuning is a bad way to teach facts: they get blurred into the weights, cannot be updated without retraining, and the model will still occasionally invent a confident variant.
• The model does not behave the way you want → prompting, then fine-tuning if prompting is not enough. Tone, structure, verbosity, consistently choosing from a fixed set of labels.
• The model cannot do the task at all → neither. That is a capability problem. Try a more capable model, or break the task into steps the model can do, or accept that this part should not use a model.

Facts go in retrieval. Behaviour goes in the prompt. Fine-tuning is for behaviour that the prompt cannot hold, and it is the third thing you try, not the first.

A decision order that works

• Write a good prompt and measure it. Not a first draft. A genuine attempt with worked examples and an explicit output contract. Most teams that conclude "prompting isn't enough" have tested one casual paragraph. Establish this number first; it is the baseline everything else has to beat.
• Add retrieval if the failures are knowledge failures. Look at what the model got wrong. If it is inventing facts, missing your documentation, or unaware of the customer's data, no amount of instruction fixes that.
• Fine-tune only if a specific behaviour still will not hold. By this point you have a measured baseline and, from the prompt-engineering work, most of the examples a training set needs.

Working in this order matters for a reason beyond cost: each step produces the material the next one needs. The examples you write while prompting become the eval set, and the eval set becomes the training set. Teams that start at fine-tuning have to build all of that anyway, just without the benefit of knowing whether it was necessary.

Where teams go wrong

Fine-tuning to teach facts. The most common and most expensive mistake. The model appears to learn your product catalogue, then confidently invents a product that does not exist, and there is no way to correct it short of retraining. Facts belong in retrieval, where they can be updated on a Tuesday afternoon.

Retrieving too much. Retrieval quality is about precision, not volume. Stuffing twenty documents into the context costs twenty documents' worth of tokens on every single request, slows the response, and often makes the answer worse, the relevant passage gets buried. Retrieving the right three beats retrieving twenty, and it is where most of the engineering effort goes.

Treating retrieval as a search problem that is already solved. Chunking strategy, what you embed versus what you return, whether keyword search should sit alongside vector search, how you handle documents that contradict each other — these are the substance of the work. "We'll use a vector database" is the beginning of the design, not the end of it.

Fine-tuning against a moving target. If the desired behaviour is still being argued about, a fine-tune freezes an opinion that will change next month, and every change costs another training run.

The cost comparison nobody runs

Compare these on the axis that bites, cost per request in production, not cost to build:

• Prompting adds tokens to every request. A long system prompt with six examples is paid for on every call, forever. It is cheap to change and not always cheap to run.
• Retrieval adds a search operation plus the retrieved tokens to every request, and a pipeline that has to be kept in sync as your data changes. The ongoing cost is real and it is mostly maintenance.
• Fine-tuning is expensive once and can be cheaper per request afterwards, because the behaviour no longer needs explaining in the prompt. Shorter prompts, fewer tokens. At high volume this matters.

Which reverses the usual intuition: fine-tuning is the expensive option at low volume and can be the economical one at high volume. If you are pre-launch and unsure of your traffic, that is another argument for not fine-tuning yet. We go into the arithmetic in more depth in shipping AI features without wrecking your unit economics.

They are not exclusive

The strongest production systems we have built use two or three together, each doing the job it is suited to: a fine-tuned model that reliably emits the right structure, fed retrieved context so its facts are current, with a prompt handling the per-request specifics.

That combination is worth building towards, not starting from. Each layer added before it was needed is a layer of cost and maintenance bought on speculation.

Triaging your own failures

The sorting exercise sounds abstract until you do it, so here is the procedure we run. Take fifty real failures from your logs and put each into one of five buckets.

It stated something false about your domain. A product that does not exist, a policy you do not have, a price that is wrong. Knowledge failure: retrieval.

It was correct but drew on nothing specific. Generic advice where the customer's own data should have informed the answer. Also retrieval: the material was never in front of it.

It was accurate but wrongly shaped. Right content, wrong length, wrong tone, wrong format, ignored an instruction it was given. Behaviour failure. Prompt first.

It was inconsistent. The same input handled well on Monday and badly on Tuesday. Also behaviour, and the strongest signal that a fine-tune might eventually pay: consistency is exactly what training a model on many examples buys you.

It could not do the task. Reasoning it could not follow, a calculation it got wrong, a judgement beyond it. Neither option fixes this. Change model, decompose the task, or remove the model from that step.

Two things usually become clear immediately. The buckets are rarely evenly filled, one dominates, and that is your answer. And the fifth bucket is normally larger than anyone expected, which is the useful surprise: a meaningful share of "the AI is bad" turns out to be a task that was never a good fit for a model.

Do this before the architecture meeting, not after. It takes an afternoon and it usually ends the debate, because the evidence is not really arguable once it is in front of everyone.

What changing your mind costs

Worth knowing before you commit, because the three options are not equally reversible.

Prompt changes are free to undo — revert the string. Retrieval is a genuine system with data to keep in sync, but it sits alongside your application and can be switched off without touching the rest. A fine-tuned model is the most entangling: it becomes a dependency you host or pay for, it has to be retrained when requirements move, and reverting to a base model means restoring all the behaviour the fine-tune was carrying in your prompt.

That asymmetry is a reason to work up the ladder rather than down it, independent of cost. The cheap options are also the ones you can abandon cheaply.

The test that settles the argument

If your team is still debating this in a meeting, the debate is the symptom. The question is not answerable by discussion: it is answerable by the triage above. Whichever bucket is fullest tells you what to build.

If you cannot produce fifty real failures, you do not have an evaluation harness, and that is the actual next task. Everything in this article depends on being able to measure whether a change helped, which is why our AI and machine learning work treats the harness as the first deliverable rather than a later hardening step.