Teams building an LLM feature almost always ask "should we fine-tune?" before they've established whether the model even needs new information or a new behavior. That question carries a heavy price tag before anyone has written a single eval, and it's usually the wrong first question.
There are three separate levers available, and they fix three different failure modes. What you tell the model in the system prompt. What you show it at query time through retrieval. What you bake permanently into its weights through fine-tuning. Mixing them up wastes weeks and a training budget that a better paragraph of instructions would have solved for free.
We've built retrieval pipelines feeding tens of thousands of embedded chunks into production chat, and we've built agent tool layers that failed for reasons that had nothing to do with the model and everything to do with the instructions around it. The pattern is consistent enough to write down as a decision rule.
The three levers solve different problems
A prompt changes behavior: tone, output format, which tool to call, how to handle an edge case. It's cheap to iterate, it costs nothing until you run it, and you can change it in a text file five minutes before a demo. Retrieval changes what the model knows at the moment of the call: your inventory levels, this tenant's contract terms, last night's OCR output. It doesn't change how the model reasons, only what facts it has in front of it. Fine-tuning changes the model itself: it shifts the default behavior baked into the weights, so the model does the new thing even with a short or absent prompt.
The mistake we see most often is reaching for the third lever to solve a problem that belongs to the first. A model that ignores your formatting instructions half the time doesn't need retraining. It needs an instruction that isn't ambiguous, or a smaller, more literal output schema.
Start with the prompt, always
Before touching anything else, rewrite the prompt and re-test against real failure cases. This sounds obvious and gets skipped constantly because prompt work feels unserious next to a training pipeline. It isn't. Most agent failures we've diagnosed on client work traced back to underspecified tool descriptions or missing constraints in the system message, not model capability. We wrote up the mechanics of that specific failure in why agents pick the wrong tool.
The discipline that makes this lever actually work is having a fixed set of test cases you re-run after every prompt change, so you know a fix for one case didn't break three others. If you don't have a labelled dataset for that yet, that's a solvable problem on its own, and we've written about building an eval set with no labelled data and no time to get there fast.
Retrieval when the model doesn't know the answer
Reach for retrieval when the failure is a knowledge gap, not a behavior gap. The model reasons fine, but it doesn't have the fact: this tenant's return policy, the spec on page 40 of a 200-page plan set, a price that changed an hour ago. No amount of prompt tuning fixes a model that has never seen the document.
We've run OCR and document pipelines feeding a vector store of roughly 21,000 embedded chunks for in-app retrieval and chat, and the lesson that held up every time was that retrieval quality lives or dies on how the source documents were chunked, not on which embedding model you picked. We covered that in detail in retrieval quality as a chunking problem and in the more mechanical chunking guide for invoices, plans and contracts.
Retrieval is also the right lever when the underlying facts change faster than you could retrain a model. Inventory counts, order status, per-tenant configuration: none of that belongs in weights. It belongs in a query that runs at request time.
Fine-tuning when behavior itself has to change at scale
Fine-tuning earns its cost in a narrower set of cases than most teams assume: high call volume on a narrow, repeated task where a smaller or cheaper model can match a bigger one once it's specialized, or a behavior pattern that a prompt genuinely cannot pin down no matter how it's worded. Style consistency across tens of thousands of generations is a fine-tuning problem. A single stubborn edge case is not.
The clearest example from our own work is not a text fine-tune at all: the vision-model quantity takeoff we built for BuildAEC reads raw construction plans and produces line-item quantities with a per-line confidence score directing human review. That task is narrow, repeats across thousands of plan sets, and general-purpose prompting a stock vision model could not get to production-grade accuracy. Specialization was the right call there because the task was constrained enough to specialize for and valuable enough to be worth the ongoing retraining cost.
Fine-tuning is a commitment to retrain on a cadence, not a one-time fix you apply and forget.
That's the honest tradeoff. A fine-tuned model locks you to a base model version. When the vendor deprecates it or ships a better successor, you retrain or fall behind, and retraining costs real time and money every cycle, not just once. A prompt edit costs five minutes. A fine-tune costs a data prep pass, a training run, an eval pass, and a repeat of all three the next time the base model changes underneath you.
A decision rule you can run in an afternoon
When a feature is misbehaving, work through this sequence before committing to any infrastructure:
- Write down five to ten concrete failure examples, not a vague description of "it's not working."
- For each one, ask: does the model lack a fact, or does it have the fact and do the wrong thing with it?
- If it lacks a fact, that's a retrieval problem. Check what context is actually being passed at call time before building anything new.
- If it has the fact and misuses it, rewrite the prompt and re-test against your full failure set, not just the one case you just fixed.
- Only after prompt iteration plateaus across your failure set, and only if call volume is high enough to justify it, treat fine-tuning as a candidate.
- Price it honestly: training and retraining cost against the added token and latency cost of a longer prompt, over the volume you actually run. Our post on predictable AI spend with wallets and per-tenant budgets covers how to model that cost per tenant before you commit.
In our experience the great majority of reported "the model is wrong" tickets resolve at step 4. Fine-tuning candidates that survive to step 6 are rare, and that's the correct outcome, not a failure to find one.
Where this rule breaks down
The sequence above assumes you can iterate on the prompt and retrieval layer freely. That assumption fails in a couple of specific situations. If the output has to conform to a rigid, proprietary schema that never appears anywhere in the base model's training data and a wrong token breaks a downstream parser, prompting alone will get you close but not reliably exact at volume. In that case, constrained decoding or a grammar-restricted output format is worth trying before a fine-tune, since it fixes the exact failure mode without the retraining commitment.
And if a compliance requirement forbids sending certain data to an external context window at all, retrieval against an outside vector store may be off the table regardless of how well it would otherwise perform, which pushes some teams toward fine-tuning or a self-hosted model earlier than the cost math alone would justify. That's a constraint from outside the technical decision, and it's worth naming before an eval process concludes something the compliance team will veto anyway.
None of these three levers is free, but they are not equally expensive, and the cheap one solves most of what gets escalated as a model problem. If you're scoping an AI feature and aren't sure which lever it needs, that's a conversation worth having before any infrastructure gets built. See what we build or start a project if you want a second opinion on the architecture before you commit to a training run.
Frequently asked
Do I need to fine-tune a model to use retrieval augmented generation?
No. Retrieval and fine-tuning are independent decisions. RAG changes what facts the model sees at call time; fine-tuning changes the model's default behavior. Most production systems use retrieval on top of a general-purpose model with no fine-tuning at all.
How much data do I need before fine-tuning is worth it?
Less about a fixed row count and more about call volume and repetition. Fine-tuning earns its cost when you're running a narrow, repeated task thousands of times a day and a smaller specialized model can match a bigger general one, or when a prompt genuinely cannot pin down the behavior you need.
Can I combine fine-tuning and retrieval?
Yes, and it's common: a fine-tuned model that's better at using retrieved context in a specific format, for example. But build and evaluate them as separate levers first so you know which one is actually responsible when quality changes.
Is prompt engineering a real substitute for fine-tuning?
For most reported failures, yes. If the model has the right facts and still produces the wrong output, that's almost always fixable with a clearer, more constrained prompt and a test set to check the fix against, not a training run.
What's the fastest way to tell which lever a failure needs?
Take five to ten real failure examples and ask, for each one, whether the model lacked a fact or had the fact and misused it. The first points to retrieval, the second points to a prompt fix. Fine-tuning only becomes relevant if a prompt fix fails to hold across the whole failure set at real volume.