Cost engineering — routing, cascades, caching economics

The 4× bill

The promo settles, latency is back under SLA — and finance forwards you the inference invoice, which is 4× what you projected. Throughput tuning from Chapter 3 helped at the margin, but cost-per-token improvements are a fraction when the real problem is that every query, easy or hard, is hitting the frontier model, and a chunk of them are near-duplicates you’re paying to answer twice.

The reflex — “switch everything to the cheap model” — would cut the bill by 94% and quietly wreck the product, because the cheap model is wrong on exactly the hard queries that made the feature worth building. Cost engineering is the discipline of cutting the bill without paying for it in accuracy, and the interview tell is whether you reach for a cost number alone or a cost number with a quality floor attached.

Predict, before reading on: frontier-only is your baseline. If you route the easy majority to a cheap model, what single property of your routing decision determines whether you saved money or just degraded quality?

The bill and its levers

The bill is arithmetic. One call costs input and output tokens at the model’s per-million rates, and the monthly bill is that times volume — mini_rag.budget again:

def api_cost_usd(input_tokens, output_tokens, rate_in_per_m, rate_out_per_m):
    return (input_tokens * rate_in_per_m + output_tokens * rate_out_per_m) / 1e6

For the bot (1,500 input, 200 output, 10K queries/day), a mini-tier model at $0.15/$0.60 per MTok costs ~$0.0003/query → ~$103/month; a frontier model at $2.50/$10.00 costs ~$0.0058/query → ~$1,725/month. That ~17× spread between tiers is the whole opportunity. Three levers act on the bill:

• Token reduction — shorter answers, fewer retrieved chunks, prompt caching. This is the Chapter 2–3 link: capping output also raised capacity, so it cuts latency and cost. Real, but bounded — you still pay frontier rates on every query.
• The cascade — route the easy majority to the cheap model and escalate only the hard queries to the frontier one. The single biggest lever, because it attacks the rate, not the token count.
• The cache — don’t call any model for a question you’ve already answered. A response cache discounts the entire bill by its hit rate.

The cascade is where the leverage and the danger both live, so it gets the next section.

A cascade is only as good as its routing signal

A cascade needs a per-query decision: handle this on the cheap model, or escalate? You route on a confidence signal against a threshold, and mini_rag.budget blends the cost:

def route_cheap(confidence, threshold):
    return confidence >= threshold        # keep cheap if confident; else escalate

def cascade_cost_usd(easy_fraction, cost_easy, cost_hard):
    """Blended per-query cost: the cheap fraction at the cheap rate, the rest escalated."""
    return easy_fraction * cost_easy + (1 - easy_fraction) * cost_hard

Route 70% to the cheap model and the blended cost drops toward the cheap tier — if the 70% you kept are genuinely the easy ones. That “if” is the entire chapter. The savings are only real when the confidence signal is calibrated — when high confidence actually means the cheap model is right. Route on a signal that doesn’t track correctness and you keep the hard queries on the cheap model precisely because it’s confidently wrong about them: all of the savings, none of the quality.

The diagnostic is escalation precision — of the queries you escalated, how many the cheap model would actually have gotten wrong. A calibrated signal escalates the right ones (high precision); a miscalibrated one escalates noise (low precision), paying for frontier calls it didn’t need while leaving hard queries on the cheap model. Calibration was the Evaluation guide’s subject; here it’s the hinge of a six-figure cost decision.

Cost with a floor

So the cost decision is never “what’s cheapest” — it’s “what’s cheapest while holding accuracy at or above a stated floor.” That makes cost engineering a gated change, the same shape as Chapter 1’s deploy:

• State the floor. “Accuracy within 1 point of frontier-only” is a number you commit to before you start, not a result you accept after.
• Verify against it. Score the cascade (and the cache) on the golden set, segmented into kept-cheap vs escalated; promote only if the floor holds. A cache adds a second risk to verify — staleness: a cached answer to a question whose underlying policy changed is confidently wrong (the same drift that bites in Chapter 7), so caches need a TTL and invalidation, not just a hit-rate target.

def effective_cost_usd(cost_per_query, cache_hit_rate, cached_cost=0.0):
    """A 30% hit rate is a 30% discount — but only if the cached answers are still right."""
    return cache_hit_rate * cached_cost + (1 - cache_hit_rate) * cost_per_query

The senior sentence is two-sided: “the cascade cut the bill ~70% and held golden-set accuracy within a point of frontier; the cache adds ~30% on top with a TTL so stale policies expire.” Cost and floor, always together.

See it: the cascade’s two curves

Frontier-only is the $1,725/month, high-accuracy baseline; all-cheap is $103 and much worse. Predict whether a cascade can cut the bill in half while holding accuracy near frontier — then flip the routing signal from calibrated to miscalibrated and watch the same threshold fall apart.

Read the gap: with a calibrated signal you hold ~frontier accuracy at a fraction of the cost and escalations are almost all warranted; with a miscalibrated one, every threshold pays more for less, and escalation precision collapses. Same cascade, same prices — the only difference is whether the routing signal tracks correctness.

How this is graded

• Technical Correctness — you decompose the bill (per-query cost × volume), know the cascade attacks the rate while token cuts attack the count, and can do the cascade-cost arithmetic.
• Trade-off Awareness — you treat cost as a trade against accuracy and latency, not a number to minimize, and you know the cache trades freshness for cost.
• Evaluation Rigor — this chapter’s weighted dimension: you attach an accuracy floor to every cost claim and verify it on the golden set, and you read escalation precision to know whether the routing signal is calibrated.
• Communication — “the cascade cut the bill 70% and held accuracy within a point; here’s the escalation precision that proves the signal is calibrated” is the two-sided sentence that signals seniority.

Industry variation

• Startups — a hand-tuned threshold and a simple exact-match cache is often the whole cost program, and it’s enough; the discipline is stating the floor, not the tooling.
• Enterprise / high volume — routing is a tuned classifier with cost-per-query and accuracy as tracked, owned metrics; caches are layered (exact then semantic) with invalidation tied to content changes.
• Regulated — a cached or cheap-model answer may carry compliance implications; “which model answered, from which cache, how fresh” is auditable.
• AI-assisted coding interviews — asked to cut cost, the signal is naming the floor and the verification in the same breath as the cascade — never “just use the cheap one.”

Stretch: you can’t run any of these loops blind

Deploy, latency, throughput, cost — every decision in this block assumed you could see what the system did: the canary’s golden-set score, the p99 distribution, the batch utilization, the cascade’s escalation precision and the cache’s hit rate. None of those are visible by default. A cascade silently slipping to a miscalibrated signal, a cache serving stale answers, p99 creeping back up — you only know if the system is emitting the right signals and something is watching them. So the loop turns from serving to observing: what do you instrument, and how do you see the one dimension infra telemetry can’t? (That’s Chapter 5 — the observability stack, and the start of the evaluate half of the loop.)