RAG end-to-end: retrieve, assemble, ground, generate

Retrieval was perfect. The answer was still wrong.

Chapter 3 closed with this trace, and now you get to resolve it. A user asks “Can I get a refund on a gift card I bought five weeks ago?” and your assistant answers “Yes — within 30 days”: wrong twice. You check retrieval — it worked. The 30-day rule ranked near the top; the gift-card exclusion was retrieved too, near the bottom of the list. Recall on the golden set: unchanged. Chunking: coherence 1.0. The model: same model that aces every other question.

So you read the assembled context — the thing the model actually received — and there it is. Your assembly step packs retrieved chunks into a fixed word budget, in rank order, until they stop fitting. The bottom-ranked hit didn’t fit. The exclusion was retrieved, scored, and then silently dropped by four lines of glue code nobody had looked at since the prototype.

Predict: which stage of your pipeline would you instrument first, and what would you log to catch this class of bug before users do?

The pipeline is code you own

RAG exists because models have a knowledge cutoff and an imagination: grounding answers in retrieved documents fixes staleness and squeezes out fabrication. The mechanism is five stages — and in mini_rag.pipeline each one returns its intermediate state, because the intermediates are where the bugs live:

class RagPipeline:
    def __init__(self, chunks, k=4, budget_words=120, min_score=0.0):
        self.index = TfidfIndex(chunks)
        ...

    def run(self, question: str) -> RagTrace:
        hits = [h for h in self.index.search(question, k=self.k)
                if h.score >= self.min_score]              # retrieve (+ floor)
        included, excluded = assemble_context(hits, self.budget_words)
        context = [h.doc for h in included]
        prompt = build_prompt(question, context)            # ground
        answer = extractive_answer(question, context)       # generate
        return RagTrace(question, hits, included, excluded, prompt, answer)

assemble_context is the four lines from the opener — include hits in rank order while they fit the budget, exclude the rest — except here the exclusions are returned, not swallowed. The RagTrace is the chapter’s real deliverable: when an answer is wrong, the trace answers “which stage?” mechanically. Empty hits → retrieval. The needed chunk sitting in excluded → context window. Everything present and the answer still wrong → look at the generator.

Notice the pipeline has three different knobs that beginners conflate: k (how many chunks retrieval may return), budget_words (how many survive into the context), and min_score (the similarity floor). They fail differently: a small k loses facts before ranking ends, a small budget loses them after, and a missing floor lets junk in — the next section makes that one bite.

Grounding: removing the freedom to improvise

The prompt is Chapter 1’s spec applied to RAG — every clause deletes a degree of freedom:

def build_prompt(question, context_chunks):
    numbered = "\n".join(f"[{i+1}] {c}" for i, c in enumerate(context_chunks))
    return (
        "You are a support assistant. Answer ONLY from the numbered context "
        "chunks below; if the answer isn't there, say \"I don't know\". "
        "Cite the chunk you used, like [2].\n\n"
        f"Context:\n{numbered}\n\nQuestion: {question}\n\n"
        "Answer (one or two sentences, with citation):"
    )

Answer ONLY from the context blocks parametric memory; the I-don’t-know rule makes missing context a safe failure instead of an invented answer; the citation makes every claim checkable after the fact.

The companion’s generator makes the point by contrast. extractive_answer indexes the sentences of the assembled context and quotes the single best match — retrieval again, one level down. It is faithful by construction: it cannot say anything the context doesn’t contain. A real LLM is the opposite — it synthesizes fluently across chunks (which you want) and just as fluently beyond them (which you don’t). That asymmetry is exactly why the grounding rules exist for real models, and why faithfulness gets measured rather than assumed — Chapter 5’s job.

Two numbers in this pipeline are policies, not defaults. The abstain threshold in the generator and the similarity floor in retrieval both draw the line between “answer” and “I don’t know” — and both must be calibrated against labeled examples, because shared filler words give junk real, nonzero scores. You’ll watch an uncalibrated pipeline confidently quote boilerplate in the demo below.

The four failure points

Every wrong RAG answer traces to one of four places. Learn the symptoms and the pipe debugs itself:

| # | Failure point | Symptom | First fixes | |---|---|---|---| | 1 | Retrieval | context is irrelevant or empty; answer says “based on the context” about the wrong thing | chunking, hybrid search, query expansion, the floor | | 2 | Context window | the right chunk was retrieved but the answer misses it | rerank before assembly, raise/spend the budget smarter, fewer-but-better chunks | | 3 | Answer extraction | the right chunk is in the context; the answer still misses the detail | tighter instructions (“extract specifically…”), citations, chain-of-thought | | 4 | Generation | the answer contradicts or invents beyond the context | grounding rules, lower temperature, few-shot “not found” examples |

Two disciplines turn the table into practice. Diagnose before fixing — a “30% wrong answers” system is almost never one cause; measure each stage (retrieval precision, coverage, groundedness) and read the numbers together like a dashboard, not a scorecard. And fix upstream first — practitioner postmortems consistently attribute the majority of RAG failures to retrieval, usually wearing a generation costume: if the right document never reached the context, no prompt engineering can save the answer, and an upstream fix cascades (better retrieval reduces hallucination and vagueness downstream).

See it: one pipeline, three traces

Each scenario below is a real RagPipeline.run() over the same chunked policy document. Predict before revealing, then toggle the knob that matters: the similarity floor in the retrieval-failure scenario, the word budget in the context-window one. Open the exact prompt — that string is what the generator actually saw, and it settles every argument about “what the model knew.”

How this is graded

• Technical Correctness — this chapter’s weighted dimension: you can walk the pipeline stage by stage, name what k, the budget, and the floor each control, and attribute a wrong answer to a stage using the trace.
• Trade-off Awareness — abstaining vs answering junk, budget size vs dilution and cost, floor height vs false abstentions — each line has a price on both sides.
• Evaluation Rigor — you diagnose with per-stage measurements before fixing, and you treat thresholds as calibrated choices, not defaults.
• Communication — “the exclusion chunk was retrieved at rank #3 and dropped by the 60-word budget; I’d rerank before assembly and re-run the golden set” beats “the model hallucinated.”

Industry variation

• Startups — the trace is the observability story: log retrieved ids, scores, included/excluded, and the prompt hash from day one; it costs an afternoon and pays for itself at the first incident.
• Enterprise — citations are mandatory, not pedagogical: auditors ask “which document said this?”, grounded prompts are reviewed artifacts, and an uncited answer is a compliance bug even when it’s correct.
• High-volume consumer products — the budget knob is simultaneously a cost and latency lever (every context word is paid tokens); teams tune budget/k/floor against answer quality and unit economics — that trade-off becomes Chapter 7.

Stretch: which pipeline is better?

You fix the opener’s bug two ways: pipeline A keeps the original tight budget and no floor; pipeline B adds a similarity floor and triples the budget. Both pass your five spot-checked questions. They disagree on cost, on safety, and — you suspect — on quality, but “I read ten answers and B felt better” won’t survive review. Design the decision: what labeled data, which metrics for each half of the system (the retrieval half and the generation half), and what would make the comparison fair? That’s Chapter 5 — evaluating RAG with golden sets, ranked-list metrics, and faithfulness — where this guide shakes hands with the Evaluation guide.