Prompt, RAG, or fine-tune: the judgment chapter

The meeting you can’t vibe through

Chapter 9 ended in the CTO’s question, so here it is again, with the budget attached: “Two years of support transcripts. Fine-tune our own model and delete half this machinery — the retrieval stack, the prompt versioning, the reranker. One model that just knows our business. Why not?”

Around the table: the platform lead nodding (fewer moving parts), finance nodding (the API bill from Chapter 7), and you — holding everything this guide has built so far. “Fine-tuning is hard” loses this meeting. So does “sure, let’s try it.” Predict: what’s true in the CTO’s proposal, what’s false, and what single distinction separates the two?

Behavior versus knowledge

One sentence resolves most of these debates: fine-tuning teaches the model HOW to respond; retrieval supplies WHAT to respond with. They’re complements, not competitors — which is why “fine-tune or RAG” is usually a malformed question, and the best production systems run both: a model tuned to the house style and output contracts, grounded at answer time by retrieval over the current documents.

Run the CTO’s transcripts through the split and the proposal falls into two clean halves. The transcripts encode style — how good agents phrase things, de-escalate, structure a resolution — and that half is genuinely trainable. They also encode policy facts as they stood at the time of each ticket — and that half is a contamination risk, not an asset: a model trained on them will cite superseded refund windows with total confidence, and no retrieval stack will be there to correct it. (Also: two years of transcripts are full of customer PII, which now lives in your weights. Scrubbing training data is real work that the proposal’s budget didn’t include.)

The comparison table, with the dimension that decides each row:

| | Prompting | RAG | Fine-tuning | |---|---|---|---| | Data needed | 0–20 examples | documents | 1K–100K examples | | Changes what | instructions | available knowledge | weights/behavior | | Knowledge updates | instant (edit prompt) | re-index | retrain | | Latency | baseline | + retrieval | can reduce (shorter prompts, smaller model) | | Best at | general tasks, prototyping | dynamic facts, grounding, citations | format, tone, vocabulary, distillation |

The decision funnel

The technique is mostly picked by what data you actually have — read the funnel top-down and stop at the first floor you can afford:

1. No data, or a handful of examples → prompting. Five great exemplars are a few-shot prompt, not a training set; tuning on them teaches noise.
2. Documents that answer your users’ questions → RAG. You built this in Chapters 2–7; it ships in days and updates by re-indexing.
3. 1K–10K+ clean labeled examples and a behavioral failure (format violations, tone, domain phrasing) → supervised fine-tuning. LoRA/QLoRA make the mechanics cheap — adapters under 1% of weights, a 7–8B model tunable on a single modest GPU — which moved the bottleneck from compute to data quality and evaluation, where it now stays.
4. 5K+ preference pairs (this answer beats that one) → preference tuning (DPO-family). Alignment to taste, not just imitation — and it amplifies whatever your annotators systematically believe, for better or worse.

Two cases cut across the funnel. Distillation is fine-tuning argued from Chapter 7’s spreadsheet: a frontier model is nailing a narrow task at painful unit economics, you’ve logged millions of its input→output pairs, and a small model SFT’d on them recovers most of the quality at a fraction of the cost and latency. No new capability — just your task’s distribution, made cheap. And hybrid is the production norm, not a compromise: tuned behavior, retrieved facts, prompted task instructions — each layer doing the one job it’s best at.

Prove it: gates and the real bill

A fine-tune is a config change with weights, so Chapter 5’s discipline applies, scaled up. Before one ships, it passes three gates:

1. Domain lift — does it beat the prompted/RAG baseline on your golden set? If not, stop here; you just saved a quarter of maintenance.
2. Forgetting check — did general capability survive? Catastrophic forgetting is the classic silent regression: the model gets your format right and quietly gets worse at everything else. Hold out a general-purpose eval and require parity.
3. Quality bar — does it meet the absolute standard the product needs, not just “better than before”?

And answer the CTO with the whole bill, because the training run is the cheap line: data curation and PII scrubbing (the actual work), eval-set construction for all three gates, a retraining cadence (every policy change that RAG absorbed by re-indexing now has a training-pipeline SLA), serving ownership if you self-host the tuned model (Chapter 7’s break-even math, now mandatory), and the option you give up — provider model upgrades no longer arrive for free; your adapter is married to a base model version. Sometimes the bill is still worth it — distillation at volume usually is. The judgment is knowing which line items dominate for this proposal.

See it: five calls to make

Each scenario below is a decision you’ll face more or less verbatim. Commit before revealing — the graded skill is the justification, and notice how often the right answer splits the proposal rather than accepting or rejecting it whole.

How this is graded

• Technical Correctness — you state the behavior/knowledge split precisely and know what each technique physically changes (prompt text, index contents, weights).
• Trade-off Awareness — this chapter’s weighted dimension: the funnel run on the data that exists, the full bill named (curation, gates, cadence, serving, foregone upgrades), and the hybrid default.
• Evaluation Rigor — no fine-tune ships without the three gates against the same golden set every other config change faces.
• Communication — “split the claim” beats “yes” and beats “no”: the CTO-answer pattern — what’s right, what’s wrong, the experiment, the bill — is the deliverable.

Industry variation

• Startups — prompting and RAG until the funnel forces otherwise; the main legitimate early fine-tune is distillation once unit economics bite at scale.
• Enterprise & regulated — data residency can make tuned self-hosted models attractive, but PII-in-weights is a governance problem prompting never had; legal reviews training data now.
• Frontier-adjacent product teams — preference tuning on real user feedback is the moat-building move, and annotator-bias amplification is its documented failure mode; the eval gates get more rigorous as the technique gets more powerful.

Stretch: all of it, at once, on a whiteboard

You now hold every layer’s judgment: prompts as specs, retrieval and its upgrades, eval as the instrument, production economics, agents and their guards, and this chapter’s adaptation funnel. The interview that tests all of it at once is forty-five minutes and one sentence long: “Design a customer-support AI for a mid-size bank.” Where do you start — and in what order do you spend the minutes so the judgment shows? That’s Chapter 11: the system-design capstone, run exactly like the real thing.