The observability stack — metrics, logs, traces, and evals

Flying blind

Look back at the last block. The cascade only saves money if it stays calibrated; the cache only helps if its answers stay fresh; the latency fix only holds if p99 actually recovered. Every one of those is a claim about what the running system is doing — and you have no way to check any of them without instrumentation. A latency dashboard might show p99 climbing again, but it won’t tell you whether the cause is the rerank stage, a cache miss storm, or one pathological 2,000-token answer. “Something is slow” is not a diagnosis.

So before the evaluate half of the loop, the observe stage: what signals do you emit, and which question does each one answer? Get this wrong and you have dashboards that tell you a problem exists and nothing that tells you what it is — or, worse, dashboards that stay green while the answers rot.

Predict, before reading on: your p99 latency alarm just fired. Which signal tells you which stage of which request burned the time — and is it the same signal that told you p99 was high?

The four pillars

Classical observability has three pillars; AI serving needs a fourth. Each answers a different question, and the skill is knowing which:

| Pillar | Answers | Shape | | --- | --- | --- | | Metrics | How often / how much? — p99, error rate, QPS, cache hit rate | aggregated numbers over time | | Logs | What exactly happened? — this request’s inputs, the retrieved chunks, the error | discrete, timestamped events | | Traces | Where did the time/decision go? — per-request causal breakdown across stages | a tree of timed spans | | Evals | Was the answer right? — groundedness, correctness, refusal behavior | scored samples of outputs |

The first three are blind to the dimension Chapter 0 was built around. A request can be counted (metric), logged (log), and traced (trace) — fast, 200 OK, every stage healthy — and still be wrong, and nothing in the classic three pillars scores the answer. Evals are the fourth pillar: the only one that watches quality, and the one this guide’s evaluate stage (Chapters 6–7) is built on. “We have observability” without evals means “we can see everything except whether it works.”

Reading a trace

The trace is the pillar that turns “p99 is high” into “rerank is the problem.” A trace is a tree of spans — timed units of work carrying attributes — and the reading that matters is self-time: a span’s own work, excluding its children. mini_prod.trace makes it mechanical:

def self_time_ms(span):
    """Exclusive time — duration minus what the children account for. A parent that is
    nothing but its children is a wrapper with ~0 self-time, however large its total."""
    return span.duration_ms - sum(c.duration_ms for c in span.children)

def rollup_by_attribute(root, key):
    """Sum self-time grouped by an attribute (e.g. 'stage') across the whole trace —
    this is the bridge from a trace to a metric: a metric is a rollup of spans."""
    out = {}
    for s in flatten(root):
        if key in s.attributes:
            out[s.attributes[key]] = out.get(s.attributes[key], 0.0) + self_time_ms(s)
    return out

Run it on a support-bot request and the 868 ms total decomposes: generate ≈ 86% (decode, exactly Chapter 2’s prediction), retrieve ≈ 13% — and inside retrieve, rerank dwarfs embed and search. That last fact is invisible to any metric; only the trace, with its per-stage self-time, says “optimize rerank, not embed.” The request span itself has near-zero self-time — it’s a wrapper — which is the distinction the reading turns on: a big total duration on a parent doesn’t mean the work happened there.

The right pillar for the question

Diagnosis is choosing the pillar that answers the question you actually have:

• “p99 doubled this week” → a metric saw it; a trace of a slow request says where.
• “this one user got a bizarre answer” → the log has the exact inputs and retrieved context; the trace shows whether it hit the cache or escalated.
• “is quality slipping?” → only an eval can answer — no metric, log, or trace scores correctness.

The failure mode is reaching for the wrong pillar: trying to debug a single weird answer from an aggregate metric (which averaged it away), or trying to detect quality decay from latency dashboards (wrong dimension entirely — Chapter 0’s myth). Match the pillar to the question, and remember the fourth one exists.

See it: where did the request’s time go?

868 ms total. Predict which stage owns it — and which retrieval step is the hotspot — before the waterfall reveals the per-span self-time.

The waterfall makes the lesson physical: the total is one number, but the shape — generate dominating, rerank the retrieval hotspot, the request span a near-empty wrapper — is the diagnosis. A metric could never have told you to look at rerank.

How this is graded

• Technical Correctness — this chapter’s weighted dimension: you name the four pillars, know evals are the AI-specific one, and read a trace via self-time rather than confusing a wrapper’s total with where the work happened.
• Trade-off Awareness — you weigh instrumentation cost (sampling traces, log volume, eval spend) against the visibility it buys; you don’t trace 100% of traffic by reflex.
• Evaluation Rigor — you treat “is it right?” as a measured pillar, not an assumption, and you know which questions a metric structurally cannot answer.
• Communication — “metrics told us p99 rose; the trace says it’s rerank; the eval says quality held” is the sentence that walks an interviewer through a real diagnosis.

Industry variation

• Startups — structured logs plus a few metrics get you far; the senior move is adding the eval pillar early even when the tracing UI waits.
• Enterprise / scale — full distributed tracing with sampling, log retention policies, and per-tenant dashboards; observability has a budget and an owner.
• Regulated — logs are an audit artifact (“what did the model see, what did it say, to whom”) with retention and access controls designed in.
• AI-assisted coding interviews — asked to “add observability,” the signal is naming the fourth pillar and which question each answers, not “add Datadog.”

Stretch: you named the eval pillar — now you have to run it

Evals are the fourth pillar, the only one that sees quality. But you can’t run a full judge on every one of 10,000 daily requests (Chapter 4 says what that costs), and a score computed on a sample of traffic is an estimate with error bars — so when this week’s sampled quality reads two points below last week’s, you genuinely don’t know whether something broke or you got unlucky. How do you operate the eval pillar on live traffic, and gate on it without thrashing on noise? (That’s Chapter 6 — evaluating in production.)