Self-host vs API vs hybrid — the standing judgment call

The standing decision

Nine chapters in, finance asks the obvious question: the bot is at 20,000 queries a day and the API bill is real — should we self-host the model and save money? It’s a fair question, and it’s also the one architectural decision every chapter so far quietly assumed away. Calling an API or running your own GPUs changes your latency, your cost curve, your data exposure, and — critically — which problems are yours. Self-hosting doesn’t just trade a per-query bill for a GPU bill; it hands you every serving, batching, scaling, and reliability problem in this guide.

There’s a clean number to compute, and you should compute it. But the most common mistake is stopping at the number, because cost is the factor that least often decides this — and an interviewer asking “would you self-host?” is testing whether you know that.

Predict, before reading on: a self-hosted GPU costs a fixed amount per month; an API costs per query. At what daily volume do they break even — and once you’ve computed it, what would make you choose the more expensive side anyway?

The break-even

Self-hosting is a fixed cost (the GPU, monthly, whether you serve one query or a million); an API is a variable cost (per query). They cross at a volume, and mini_rag.budget computes it:

def break_even_queries_per_day(gpu_monthly_usd, api_cost_per_query):
    """The daily volume where a fixed GPU cost equals the per-query API bill. Below it the
    API is cheaper; above it self-hosting is — all else equal, which it rarely is."""
    return gpu_monthly_usd / 30.0 / api_cost_per_query

A $3,000/month GPU against a frontier-class API at $0.006/query breaks even at 3000 ÷ 30 ÷ 0.006 ≈ 16,700 queries/day. So the bot at 20,000/day is just past it — self-host saves about $600/month ($3,000 vs $3,600). Two things this makes obvious:

• The crossover swings enormously with the API rate. Swap the frontier API for a hosted open model at $0.0006/query and the break-even jumps to ~167,000/day — because the thing you’d self-host is cheap to call too. You’re usually not choosing self-host-vs-frontier; you’re choosing self-host-an-open-model vs call-that-same-open-model.
• Near the crossover, cost barely decides. A $600/month difference is noise next to the salary cost of the engineer keeping your GPUs healthy. The arithmetic says “either,” which hands the decision to everything else.

Beyond cost

The factors the break-even ignores are usually the ones that decide:

• Latency. Self-hosting can give lower, more predictable tail latency (no shared multi-tenant queue, no network hop) — decisive for a voice or real-time product. Or it can be worse, if you can’t keep utilization high (Chapter 3). Either way it’s a serving problem you now own.
• Data residency & privacy. A regulated workload that can’t send data to a third party forces self-hosting (or a private deployment) regardless of cost. This is frequently the actual reason a company self-hosts, and it’s not on the cost axis at all.
• Control & customization. Self-hosting lets you fine-tune, pin a version forever (no provider swap — Chapter 1), and tune the serving stack. An API gives you the frontier model’s quality and the provider’s improvements for free.
• Operational burden. This is the one candidates forget: self-hosting makes Chapters 2, 3, 5, and 9 your job — capacity, batching, observability, on-call. The break-even rarely prices in the team it takes to run the thing.

The senior framing inverts the naive one: start from latency, residency, and control; use the break-even to size the cost consequence — not the other way around.

The hybrid

The decision is rarely all-or-nothing, and the answer that usually wins is a hybrid that takes the best of each — and it’s the cascade from Chapter 4 wearing a deployment hat:

• Self-host the high-volume, cheap, steady path — the easy majority of queries on an open model you run, where the volume justifies the fixed cost and the workload is predictable.
• Call an API for the rest — burst capacity above your self-hosted ceiling, frontier escalations for the hard queries (the cascade’s expensive path), and the long tail of rare request types not worth provisioning for.

This captures most of the savings (the bulk of traffic is on your cheap fixed-cost path) while letting the API absorb spikes and the hard tail — so you don’t provision GPUs for your peak, and you don’t pay frontier rates for your easy majority. It’s more moving parts, but it dominates the pure plays for most products above the smallest scale.

See it: where’s the crossover?

The default is the bot’s situation — a $3,000 GPU vs a frontier API at 20K/day, right near the line. Predict which side wins, then move the volume and the API rate and watch the crossover move — and notice how often “cheaper” is close enough that the non-cost factors should decide.

The lesson the sliders teach: the crossover is a moving target set mostly by the API rate, and near it the monthly difference is small — exactly the regime where latency, residency, and ops, not the dollar figure, make the call.

How this is graded

• Technical Correctness — you compute the break-even correctly (fixed vs per-query, the crossover volume) and know the API rate, not the GPU cost, dominates where it lands.
• Trade-off Awareness — this chapter’s weighted dimension: you subordinate cost to latency, residency, control, and ops, and you reach for the hybrid instead of a pure play.
• Evaluation Rigor — you price the operational cost of self-hosting (the team, the on-call, every prior chapter becoming yours), not just the GPU line item.
• Communication — “above the break-even on cost, but residency forces self-host, so we self-host the sensitive path and API the rest” is the multi-factor answer that signals seniority.

Industry variation

• Startups — almost always API first: you can’t afford the ops burden, and the frontier model’s quality is a feature. Self-host only when a hard constraint (cost at real scale, residency) forces it.
• Enterprise / regulated — residency and control push toward self-hosted or private deployments early; the hybrid (private core, API edges) is common.
• High volume / latency-critical — self-hosting an open model on tuned serving wins on both cost and tail latency once volume is high and steady.
• AI-assisted coding interviews — asked “self-host or API,” the signal is computing the break-even and naming residency/latency/ops as what actually decides — never cost alone.

Stretch: you have every piece — now assemble the system

Ten chapters have built the production loop one stage at a time: deploy safely, serve fast and affordably, observe, evaluate, respond, and choose your infrastructure. An interview won’t ask for the pieces one at a time — it’ll put a blank whiteboard in front of you and say “design a production AI system,” and expect you to assemble them under time pressure, leading with requirements and SLOs before boxes. Can you compose the whole loop into one coherent design, and defend the trade-offs? (That’s Chapter 11 — the system-design capstone.)