Verification under automation

The “done” that wasn’t

A candidate gets the AI’s solution, runs it on the sample input the problem provided, sees the right answer, and says “done” — eight minutes, green. The interviewer types one more input: an empty list. It throws. The candidate’s confidence evaporates, and so does the signal.

Predict, before reading on: what’s the gap between what the candidate did and verification? The sample input is the case the code is most likely to handle — it’s the one everybody, including the model, designed for. Running it proves almost nothing.

”It ran” is not “it works”

Two claims hide inside “it works,” and they’re miles apart. It ran means one input produced a plausible output once — a demo. It’s verified means the output passes the gate you preregistered: the empty case returns [] not a crash, the invariant holds across inputs, the boundary behaves. The demo is evidence about the easiest case; verification is evidence about the cases that actually fail.

Automation widens the gap in two ways. The output looks complete earlier, so the felt distance from “ran” to “done” shrinks even though the real distance hasn’t. And because you didn’t write it, you have less implicit knowledge of where it’s weak than you would for your own code — so the run you should least trust is exactly the one that feels most convincing. The discipline is to refuse “it ran” as a stopping point, every time, and go to the gate.

Verify against the gate

The gate is your preregistration, executed: run the committed tests, check each invariant, and add the cases you turned up while reading (Ch 5). It is binary — the output passes the gate or it doesn’t — and the one move that matters is refusing to negotiate with happy-path green. “It passes the example” is not a partial pass of the gate; it’s the gate untouched.

This is the loop closing on itself: prompt → review → run → confirm, iterated. You run against the gate; if it fails you go back through review and prompt with the failure in hand; you repeat until the gate is green for real. Predict the right call in each case:

The through-line: a run against the handed-down example proves the least, an independent preregistered standard proves the point, and the gap between them is where shipped-vs-demoed lives.

Debugging AI output

When the gate fails, you have a choice that separates the people who use AI well from the people it uses. The weak move is blind re-prompting — “it’s broken, please fix” — which throws away the one piece of information you just earned and lets the model rediscover (or re-miss) the problem. That’s prompt-roulette.

The strong move treats the verification result as the best debugging prompt you’ll ever have. Three steps: localize (which invariant or test failed), understand why (read it, Ch 5 — don’t fix what you don’t understand), then constrain the fix: “this fails on empty input — returns None, should return []; fix that without changing the non-empty behaviour.” You’re feeding the model the exact gap and fencing off what must not change. That’s prompt → review → run → confirm turned into a debugging loop, and it converges; blind re-prompting wanders.

How this is graded

Evaluation Rigor is this chapter’s weighted dimension — verification is rigor made operational:

• Evaluation Rigor (weighted) — you run the preregistered gate, not the handed-down example, and you treat green as meaningful only because the standard was independent of the code.
• Technical Correctness — your debugging localizes the actual failing invariant and you understand the cause before fixing, rather than mutating code until it passes.
• Trade-off Awareness — you choose how much verification the stakes warrant (a boundary test vs a property check vs an adversarial sweep) instead of all-or-nothing.
• Communication — saying “it ran on the sample, but I haven’t verified the empty case or the per-user invariant yet” is the honesty that reads as senior, not the premature “done.”

Industry variation

• Startups — verify the core invariant even when you skip the polished test harness; the one gate that prevents shipping a broken feature is non-negotiable, the rest is triage.
• Big tech — verification is automated and reviewed: AI code passes CI gates and human review, and “the gate” is institutional, not personal.
• Regulated (fintech, health) — verification is the audit record; the committed standard and its pass history document what the AI-written code was held to and by whom.
• Frontier labs — verify novel code against theory and properties, not just examples, because the “expected output” for a new method is itself uncertain.

Stretch: you can run the whole loop perfectly — in silence — and still fail

You’ve now closed the five solo stages: clarify, preregister, prompt, review, run. A candidate can do all five flawlessly and still get a no, because the sixth stage — explaining — runs through all of them and is where the most offers are actually lost. Before the next chapter: think back over the limiter you just built and verified. How much of your reasoning would an interviewer have seen if you’d done all of it without narrating?

Next: Chapter 7 — Communication while you delegate, the weighted dimension of the whole guide, and the one most strong candidates underweight.