Evaluating AI-assisted output

The thirty descriptions nobody read

A team uses AI to generate 200 product descriptions. They spot-check five, the five look great, they ship all 200. Thirty of them have a subtly wrong attribute or a claim the product can’t make. The five they read were, of course, the five most likely to be fine.

Predict, before reading on: what would have caught the thirty? Not reading more samples — you’d have to read nearly all of them to be sure. The answer is a different kind of check: a standard you can run over everything, automatically.

When reading doesn’t scale

There’s a threshold. Below it — a function, a small change — you verify by comprehension plus a handful of preregistered tests. Above it — a fourteen-file refactor, 200 generations, a feature that produces output for every user — you cannot read your way to confidence, and trying to is both infeasible and biased. The shift is from comprehension to measurement: from “I read it and it’s right” to “it passes a standard I can run over all of it.”

This is not a new idea in the series — it’s the silent-failure problem from Guide #3 (the system can be confidently wrong while looking fine) and the evaluation discipline from Guide #1 (you measure quality, you don’t assume it), now pointed at output you produced with AI and are responsible for. Working with AI at scale doesn’t escape evaluation; it requires it.

Build the gate: golden sets for AI output

The gate is a standard you can run automatically over the whole output. Its shape depends on the output:

• Deterministic code. The gate is a test suite encoding your invariants, run in CI over the whole change — your Chapter-3 preregistration grown into a standing, automated check. For a migration, it’s the tests that must pass at every one of the 30 call-sites.
• Generated artifacts (descriptions, configs, data). Build a golden set of properties: the things that must hold (no banned claims, correct attributes, length and format bounds) and run all outputs through it, reserving human reading for what the check flags.

This is the same construction Guide #1 teaches and its mini_eval companion implements: a set of cases, a way to score against them, a gate on the result. The bridge is exact — preregistration is a golden set for one unit; evaluation is a golden set for the whole output. Predict the right approach:

The pattern: at volume, build a gate you can run on everything, and spend scarce human reading on what the gate flags.

Evaluating non-deterministic output

The hardest case is when the AI’s output isn’t a fixed function but a generative feature — answers, summaries, code suggestions that vary per run. You can’t assert exact equality, because there’s no single right string. You evaluate properties and quality instead: groundedness, correctness, format, safety — measured with metrics or an LLM judge over a sampled set, exactly as Guide #1 teaches. One honest note, consistent with this series’ demo policy: you evaluate over samples and report measured quality; you don’t fabricate a single “the AI said X” output and call it verified.

This is the moment working-with-AI fully becomes LLM evaluation. If your AI-assisted work is shipping a generative feature, the skill you need is the whole of Guide #1 — and the bridge runs both ways: the loop (clarify, preregister, verify) is how you build the feature, and evaluation is how you know it works at scale.

How this is graded

Evaluation Rigor is this chapter’s weighted dimension — at scale, rigor is a runnable gate:

• Evaluation Rigor (weighted) — you build a standard you can run over all the output, and you evaluate generative output by measured quality rather than a sample or a single check.
• Technical Correctness — you know why exact-match fails for non-deterministic output and what to measure instead (properties, not strings).
• Trade-off Awareness — you size the eval to volume and risk: a property check vs a labeled set vs a judge, sample vs full.
• Communication — you can say “I can’t read 200, so here’s the gate I’d run on all of them,” which reads as senior at scale.

Industry variation

• Startups — a lightweight golden set beats no eval; build the smallest gate that catches the failure you can’t afford, and grow it as volume grows.
• Big tech — evaluation infrastructure for AI output is a core competency; this is the eval-at-scale skill that distinguishes “used AI” from “shipped AI responsibly.”
• Regulated (fintech, health) — the eval is also the audit: documented pass rates and the cases evaluated become part of the record.
• Frontier labs — evaluating model output is the job; rigorous eval design (and resisting metrics that can be gamed) is the discipline being hired for.

Stretch: evaluation tells you it’s good now — who owns it when it isn’t?

A gate tells you the output meets the standard today. It doesn’t answer the question that lands in a postmortem six weeks later: when this AI-written code breaks, who’s accountable, and how do you show what it was checked against? Before the next chapter: if a bug ships in code an agent wrote and you approved, what’s the honest answer to “how did this happen?”

Next: Chapter 11 — Provenance & accountability, where “the AI wrote it” turns out not to be an answer.