Chapters
Every chapter, grouped by Part. Use the card metadata to calibrate how much trust to place in a chapter's specific claims.
Part 1
-
Why evaluation is the scarce skill
Model training is commoditizing; knowing whether a system actually works is not. Why evaluation is the highest-leverage AI-engineering skill of 2025–26, what interviews test, and the four-dimension lens this guide grades you against.
-
The evaluation mindset
Before any metric: how is the output used, what does each error cost, and what failures do you fear? The evaluation mindset — error taxonomy before metrics — drilled on three systems where the obvious metric hides the real failure.
-
Offline metrics and the threshold trade-off
Precision, recall, F1, and the operating point — why a single accuracy number lies on imbalanced data, and how to choose a threshold from the business cost of each error. You build the metrics in mini_eval and feel the trade-off in an interactive demo.
-
Confidence intervals and statistical rigor
A metric from a finite eval set is a random quantity. Bootstrap confidence intervals, paired model comparison, and a permutation test — built in mini_eval — to answer the question behind every leaderboard: is B really better than A, or did it just win the sample?
-
Calibration and reliability
A model can rank risk perfectly and still lie about probabilities — saying 90% on a batch it gets right 60% of the time. Discrimination vs calibration, reliability diagrams, ECE and Brier built in mini_eval, and why a calibrated probability is the product when a human acts on it.
-
Data integrity: leakage, contamination, reproducibility
The fastest way to a great eval number is a broken eval set. Train/eval overlap, parametric-memory leakage in RAG, adaptive overfitting from tuning on a fixed set, benchmark contamination, and the reproducibility hygiene — frozen sets, pinned versions, seeds — that makes a number mean something.
-
Reference-based vs reference-free evaluation
Do you have a golden answer to compare against, or not? That single question routes every grading decision — exact match and verifiers when there's one right answer, embedding similarity and multiple references when many phrasings are valid, and reference-free LLM judges when there's no gold at all.
-
LLM-as-judge and its biases
Using a strong model to grade another model's outputs is the workhorse of modern evaluation — and it is biased in predictable, demonstrable ways. Position, verbosity, and self-preference bias, why they happen, and the swap-and-aggregate mitigation, built in mini_eval with no live model.
-
Benchmark literacy
MMLU, GPQA, HumanEval, SWE-bench, GAIA, Chatbot Arena, HELM, LiveBench — what each actually measures, and the three threats that turn a headline score into a lie: saturation, contamination, and construct validity. How to read a leaderboard like an engineer, not a marketer.
-
RAG evaluation
A retrieval-augmented system is two systems in a trenchcoat — a retriever and a generator — and a single end-to-end score hides which one failed. Retrieval metrics (precision@k, recall@k, NDCG) in mini_eval, faithfulness for the answer, and the Type-I vs Type-II hallucination split that tells you which half to fix.
-
Agentic and task evaluation
An agent is stochastic and multi-step, so 'did it work?' becomes 'how often, over how many tries, and did it get there the right way?' pass@k vs pass@1 built in mini_eval, why a single successful demo is an anecdote, and grading the trajectory — tool use, efficiency, cost — not just the outcome.
-
Production evaluation and monitoring
Offline eval is a snapshot; production is a stream. Why a frozen offline metric stays green while live quality drifts, the online signals and guardrails that catch it, drift detection, cost and latency as first-class metrics, and the alerting that turns evaluation into a continuous process.
-
Capstone: design an evaluation strategy
The interview question this whole guide was for: 'design an evaluation strategy for this system.' A framework that is just the guide in order — clarify the use, choose metrics, protect the data, quantify uncertainty, decompose the pipeline, and monitor in production — with the four-dimension rubric as the scorecard you're graded on.
Part 2
-
Why LLM application engineering is the scarce skill
Calling a frontier model is one line; turning that call into a system someone can rely on is the job. The LLM-app stack — model, retrieval, orchestration, evaluation — what each layer owns, why integration (not training) is the interview-tested skill, and the three shortcuts that look like the answer and aren't.
-
Prompt engineering as a discipline
Prompting isn't magic words — it's writing a spec. What to pin down (role, context, instruction, format), why anything code consumes needs a structured-output contract you validate, and why prompts are code: versioned and tested against a held-out set, not three happy-path examples.
-
Retrieval 101 — embeddings and vector search
Why retrieval exists, and how it actually works: vectorize, score by cosine, take the top k. You build the whole mechanism as a readable TF-IDF index, watch exactly where lexical matching misses (paraphrase, morphology), see what dense embeddings change — only the vectorizer — and scale the same idea honestly through ANN and vector stores chosen by constraints.
-
Chunking and document representation
Retrieval ranks chunks, not documents — so the chunker decides what a vector can even represent. Build three honest strategies (fixed, sentence, paragraph) plus overlap, see a real fact get cut in half and lost, learn why extraction quality is the ceiling and tables are first-class, and measure chunking like an engineer instead of eyeballing it.
-
RAG end-to-end: retrieve, assemble, ground, generate
Assemble the full RAG pipeline and own every stage of it: retrieval feeds a budgeted context assembly, a grounded prompt wraps the survivors, and a generator answers — or abstains. Build it with a per-stage trace, learn the four failure points (retrieval, context window, extraction, generation), and diagnose upstream-first instead of blaming the model.
-
Evaluating RAG: the builder's loop
The Evaluation guide teaches the metrics; this chapter wires them into the build. Construct a golden set from your own corpus (including should-abstain rows), evaluate the retrieval half and the generation half of your pipeline separately, and decide between configs with outcome categories instead of vibes — the instrument that turns 'B felt better' into an engineering call.
-
Advanced RAG: reranking, hybrid retrieval, and query rewriting
The upgrades that attack ranking quality itself — and the judgment to buy them in the right order. Two-stage retrieval with a reranker (never over the whole corpus), hybrid lexical+dense fusion with RRF, stemming as a vectorizer swap, query expansion through HyDE, and the routing family (Self-RAG, CRAG, Graph-RAG) — each priced in latency and complexity, each verified against Chapter 5's golden set.
-
RAG in production: latency, cost, and what to watch
The two budgets every production RAG system answers to — milliseconds and dollars — and the instrumentation that keeps it honest. Latency anatomy (TTFT vs decode, and why output length is the forgotten multiplier), token economics (caching, cascades, the self-host break-even), and observability that watches the knowledge boundary, not just the GPUs.
-
Agents and tool use: the loop that decides
An agent is a loop in which the model decides what happens next — which makes control flow, cost, and failure all stochastic. Build the loop with its guards (budgets, loop detection, errors-as-observations), learn tool design as the reliability lever, gate the writes, and earn the judgment call interviews actually probe: when an agent is the wrong answer.
-
Multi-agent orchestration: when one loop becomes a team
Multi-agent earns its cost in exactly three currencies — specialization, parallelism, and clean-context decomposition — and charges for it in coordination. Build the supervisor pattern (workers with least-privilege tools and single-ticket contexts), make failure isolation real (one worker's loop costs one handoff, not the run), and learn the coordination failures by name: deadlock, livelock, contention.
-
Prompt, RAG, or fine-tune: the judgment chapter
The #1 judgment question in AI-engineering interviews, answered the way seniors answer it: lead with when NOT to fine-tune. The behavior-vs-knowledge split, the data-availability funnel (prompt → RAG → SFT → DPO), the distillation-for-economics case, and the three evaluation gates that a fine-tune must pass before it ships — plus the costs that aren't the training run.
-
Capstone: design an LLM application, under interview conditions
Forty-five minutes, one sentence: 'Design a customer-support AI for a mid-size bank.' The capstone assembles the whole guide into the 5-step framework that drives an AI system-design interview — problem framing, architecture, deep dive, trade-offs, eval — with a worked bank-bot mock, a diagnose-the-gap drill, and the four-dimension rubric as your scorecard.
-
Interview craft and transfer: the skill above the skills
The meta-chapter: why communication outranks knowledge in documented rejection reviews, the craft loop that makes reasoning visible, the new AI-assisted coding round and the verification-under-automation skill it actually grades, and deliberate practice for transfer — mapping questions you've never seen onto the structures this guide built.
Part 3
-
Why production is where AI systems live or die
The demo worked, the offline eval passed, the dashboards are green — and the system is failing anyway. The production loop — deploy, serve, observe, evaluate, respond — what each stage owns, why AI systems fail silently in ways classical services don't, and the three post-launch myths that let them.
-
Deploying AI safely — shadow, canary, flags, fallback
The production loop starts at deploy: getting change into production — including the changes you didn't make, like a provider swapping the hosted model under you — without letting it surprise users. The four mechanisms (shadow, canary, feature flag, fallback), why a model-id change is a release and not a config tweak, and why every promotion gates on the golden set rather than on green dashboards.
-
Latency anatomy — TTFT, decode, and where time goes
A traffic spike broke p99 while per-request latency never changed — because a slow system under load is a queue, not a slow model. Where a request's time actually goes (prefill vs decode, and why you never quote latency without the output-length assumption), why latency is a distribution measured at the tail, and why capacity = serving slots / service time is the arithmetic that explains the cliff.
-
Throughput & efficiency levers — quantization, batching, speculation
Chapter 2's fix for a latency cliff was 'more concurrency' — but every serving slot holds a KV cache in VRAM, so batch size is memory-bound, not free. The KV-cache ceiling on batching, and the three levers that get more requests through a fixed GPU — quantization (free memory), continuous batching (kill idle slots), speculative decoding (more tokens per step) — each matched to the bottleneck it actually attacks, and the cost each one carries.
-
Cost engineering — routing, cascades, caching economics
Chapter 0's bot ran 4× over budget. The bill decomposes into per-query token cost times volume, and the two biggest levers aren't efficiency — they're the cascade (route the easy majority to a cheap model, escalate the rest) and the cache (don't run a model on a question you've answered). The catch the island makes visceral: a cascade only saves money if its routing signal is calibrated, and a cost number without a stated accuracy floor is not a saving.
-
The observability stack — metrics, logs, traces, and evals
Every serving decision in the last block assumed you could see what the system did — but a metric tells you p99 is high, not where in which request the time went, and none of the classic three pillars watch the one dimension that matters for an AI feature: was the answer right. The four pillars (metrics, logs, traces, evals), how to read a trace to locate where a request spent its time, and which pillar answers which question.
-
Evaluating in production — sampling, judges, and regression gates
The eval pillar, operated on live traffic. You can't score every request, so eval-in-prod is sampling — grade a fraction with the same metric you validated offline. And because you sampled, the number has error bars, so a regression gate must fire on a confidence interval (a drop beating sampling noise), not the point estimate. Plus what you sample (stratify the hard segments) and whether your judge is valid matter as much as how much.
-
Drift & quality decay — detecting the slow failure
Chapter 0's worst failure had no change to blame: groundedness slid for a month as the help center was reorganized and the query mix drifted, and every point-in-time gate stayed green. Drift is the failure a frozen offline set and a two-point regression gate both miss. The three pieces of a drift monitor — a rolling window to smooth the signal, a non-parametric rank test (Mann-Whitney) to detect a distribution shift, and an SLO guardrail to fire the alert — and how to tell drift from a regression.
-
Guardrails in production — input gates, output gates, gated actions
The observe and evaluate stages let you see a bad answer; they don't stop it reaching the user. The respond stage starts with guardrails — the real-time gates that prevent a harmful or manipulated output from going out. Why prompt injection (OWASP LLM01) is a structural failure no system prompt fully fixes, the defense-in-depth stack (input gate, output gate, gated action), and why some failure classes must be blocked at the boundary, not detected after.
-
Incidents & reliability ops — SLOs, runbooks, degradation modes
Guardrails stop the failures you anticipated; the bad day is the one you didn't. Reliability as a budget, not an absolute — an SLO with an error budget you spend on velocity, applied to AI quality and not just uptime. Degradation modes that turn an upstream outage into a graceful fallback, and runbooks that turn a 3 a.m. alert into a tested procedure: mitigate first (flip the flag, roll back), then find the cause.
-
Self-host vs API vs hybrid — the standing judgment call
Every chapter so far quietly assumed a hosted model API. The standing decision underneath: self-host the model, call an API, or run a hybrid. The break-even is real arithmetic — a fixed monthly GPU cost versus a per-query API bill, with a crossover volume — but cost is necessary, not sufficient. Latency, data residency, control, and the operational burden of owning your own serving stack (every prior chapter becomes yours) routinely decide it before volume does, and a hybrid often captures most of the savings at a fraction of the ops.
-
System-design capstone — production-grade
Ten chapters built the production loop one stage at a time; an interview hands you a blank whiteboard and 'design a production AI system' all at once. The method that composes them: lead with requirements and SLOs before architecture, derive the design from the loop's stages, defend each trade-off against a stated target, and anticipate the failure modes — naming the loop mechanism that bounds each. A worked capstone (a bank's internal policy-Q&A bot) walks the whole thing end to end.
-
Interview craft & transfer — the production story
The same production experience reads as junior or senior depending on how you tell it. The signals interviewers actually read for seniority — leading with requirements and SLOs, estimating cost and latency unprompted, raising evaluation before being asked — and the documented failure modes they counter ('no production experience', 'ignoring ops/cost/eval'). How to transfer the production loop to any system the interviewer invents, and how to rewrite a weak answer into a strong one. The closing chapter, which returns to the question that opened the guide.
Part 4
-
Why working with AI is its own engineering skill
The AI wrote the feature, it compiles, the tests pass — and the interviewer's next question ends the loop: why is it built this way, and what breaks on the input nobody tested? Working with AI is a distinct skill from coding and from prompting: the bottleneck moved from producing code to governing, verifying, and explaining code you didn't type. This chapter frames the loop the rest of the guide builds.
-
What interviewers actually test
Two candidates, the same problem, the same AI, the same correct final code — one passes, one fails. The artifact isn't the assessment. AI-assisted rounds score four things the model can't do for you: strategic clarification, comprehension, verification, and communication — which map exactly onto the series' four-dimension rubric, with the stakes raised because the typing no longer proves you understood anything.
-
Clarify before you code
The loop's first stage and its highest return. An AI assistant amplifies your aim — including when it's wrong — and never pushes back on a vague request the way a human collaborator would. So the discipline of pinning the real problem before you prompt, once shared with a teammate, is now entirely yours: a twenty-second question against an hour of confidently solving the wrong problem.
-
Preregistration: commit the target before you prompt
Borrowed from science, where you register your hypothesis and analysis before seeing the data so the data can't rationalize you. In AI-assisted work the 'data' is the model's output: decide what 'right' means — the interface, the invariants, the tests — before you prompt, so the output is gated against a standard instead of becoming the standard. This is the durable, tool-version-proof skill: governance over automation, not prompt theater.
-
Strategic prompting: the prompt is a specification
Prompting is not incantation and not the whole skill — it's the stage where your committed spec becomes a precise request. A vague prompt outsources the specification to the model's statistical defaults; a precise one hands over the invariants and constraints you preregistered. The judgment calls: decompose or one-shot, and what to constrain — including the negatives the model will otherwise fill in for you.
-
Reading AI-generated code
Comprehension is the loop stage automation can't do for you, and reading code you didn't write is harder than reading your own — you must reconstruct intent from output instead of building it as you go. Worse, AI code is fluent by construction: clean, idiomatic, well-commented whether or not it's correct, and that polish actively suppresses your skepticism. The discipline is active reading against the spec you committed, hardest exactly where the code looks cleanest.
-
Verification under automation
Executing AI code on the example you were handed is the weakest possible evidence — it's the case most likely to pass. Verification is running the output against the standard you preregistered: the boundaries, the invariants, the adversarial cases. This chapter closes the loop's solo stages — prompt, review, run, confirm — and turns a failure into a precise debugging prompt instead of prompt-roulette.
-
Communication while you delegate
The loop's sixth stage, and the one most strong candidates underweight — it runs through all the others rather than coming last. When you wrote every line, the code was a trace of your thinking; now the AI writes the lines, so the code traces its thinking and yours is invisible unless you voice it. Silent competence reads as luck. Narration is the interviewer's main window into the comprehension and verification that automation hid — which is why communication is this guide's weighted dimension and the single most common reason strong candidates are rejected.
-
Anti-patterns: failing while the code looks fine
The named failure modes of AI-assisted work are dangerous precisely because they mimic the virtues: speed looks like skill, confidence looks like comprehension, a passing demo looks like verification. Copy-paste automation, prompt theater, automation bias, and the velocity trap each feel productive and often pass the immediate check — then fail the follow-up question or the boundary input. Each is a skipped loop stage wearing a costume; telling the anti-pattern from its look-alike virtue is the skill, and the loop is the cure.
-
Agentic workflows: governing what you can't read
The loop scales from a single function to multi-file, agent-driven work — and an agent doesn't change the loop, it changes what you can hold in your head. You can no longer read every line, so governance shifts from per-line review to per-checkpoint review: scope the work into units small enough to preregister and verify, gate between them, and govern the seams, the gate, and the riskiest units rather than every line. This is what the un-one-shottable interview problems are built to test.
-
Evaluating AI-assisted output
When AI produces output at volume — code across a repo, or generations as a feature — reading samples is the happy-path trap at scale. You need a standard you can run automatically: a golden set, properties, a gate. For deterministic code that's a test suite in CI; for non-deterministic generative output it's measured quality (metrics or an LLM judge) over samples — which is the evaluation discipline of Guide #1, pointed at the AI's output you're now responsible for. Preregistration grown up into a standing eval.
-
Provenance & accountability
You can delegate the typing, not the accountability. The moment you ship or approve AI output, you own it as if you'd written it — 'the AI wrote it' is not an explanation, an excuse, or an audit answer. Provenance is how you make that ownership real: what was AI-generated, what it was reviewed and tested against, who approved it. For your own work the loop itself is your provenance — the ability to explain and defend the code; for teams and regulated systems it becomes an explicit record.
-
Interview craft & transfer: the whole loop in one room
The capstone: the whole skill run at once, under interview conditions, on a problem you haven't seen. Time pressure and an observer make the two highest-value stages — clarify/preregister and explain — the first to drop, so the craft is rehearsing the loop until it survives. A worked mock round, annotated against the four-dimension rubric, then a self-assessment you can run on your own attempt. The loop and the rubric are problem-independent — that's what transfers.