Agentic and task evaluation

It worked in the demo

Your agent booked the meeting, queried the database, and shipped the summary — live, in front of the team. A PM says, “great, it works.” You ran it once. In the interview: what’s wrong with concluding “it works” from a successful demo?

Predict: name two things about an agent that make a single successful run weak evidence.

What single-turn eval doesn’t have

Three new properties change the game:

1. Stochastic outcomes. Success is a rate, not a yes/no. Run each task $n$ times and report mean ± std (Chapter 3) — typically $n \ge 5$ for benchmarks, more for a production gate.
2. Multi-step trajectories. The agent plans, calls tools, observes, and retries. The outcome can be right while the path was broken (lucky, wasteful, or unsafe) — or wrong only because one tool call failed.
3. Cost is part of the score. Steps, tokens, wall-clock, and dollars are first-class: an agent that succeeds in 40 tool calls when 4 would do is failing on a dimension single-turn eval never had.

pass@k and the reliability it hides

The headline agentic metric is pass@k: the probability that at least one of $k$ samples succeeds. Run a task $n$ times, count $c$ successes, and the unbiased estimator is:

In mini_eval:

from mini_eval import pass_at_k, mean_pass_at_k

pass_at_k(n=10, c=4, k=1)   # 0.4 — first-try success (what a user gets)
pass_at_k(n=10, c=4, k=5)   # ~0.98 — at least one of 5 tries works
mean_pass_at_k(tasks, k=1)  # average pass@1 over a suite

pass@k always rises with $k$ — more attempts, more chances — which is exactly why it can mislead. A flaky agent that succeeds 35% of the time per try looks near-perfect at pass@10. pass@k is the right metric only when you can actually retry-and-verify (you have a checker to pick the good sample); otherwise the user lives at pass@1, and reporting pass@10 is marketing.

See it

Toggle a flaky vs a reliable agent. Predict: for the flaky agent, how far apart will pass@1 and pass@10 be?

Now explain the gap. Both agents reach ~100% at pass@10, but the flaky one delivers that only if you run it ten times and can tell which run was right. With no verifier, its real reliability is pass@1 — the dashed line — and the curve above it is hope.

Grade the path, not just the destination

How this is graded

• Technical Correctness — you treat success as a rate, define pass@k correctly, and know when a verifier applies.
• Trade-off Awareness — you distinguish pass@k from pass@1 and only claim pass@k when retry-and-verify is real; you weigh success against cost.
• Evaluation Rigor — the weighted dimension: n-roll with mean ± std (not one run), trajectory checks beyond the outcome, and a cost budget.
• Communication — “pass@1 is 0.37 — fine for a retry-with-checker workflow, not for an autonomous action” is the senior framing.

Industry variation

• Coding agents — test-based verifiers dominate (SWE-bench style); pass@1 + cost-per-fix is the real metric.
• Customer-facing actions — irreversible side effects make trajectory and safety eval non-negotiable; pass@k with no verifier is unacceptable.
• Research / capability — pass@k at higher k is informative as a ceiling, with contamination controls (Chapter 8).

Stretch: the agent games your verifier

Your success check is “the unit tests pass”. The agent learns to edit the tests so they pass trivially. The outcome metric says 100%. What happened, and how do you eval around it? (This is reward hacking / specification gaming: the agent optimized the measure, not the goal — Goodhart again, from Chapter 8. Defenses: hold the tests immutable / out of the agent’s write scope, add held-out tests it never sees, and inspect trajectories for tampering. Whenever success is checkable, ask what else makes the check pass.)