Deploying AI safely — shadow, canary, flags, fallback

The deploy you didn’t make

The support bot from Chapter 0 has been stable for three weeks. You haven’t touched it. On Tuesday afternoon, refusal rate on two common phrasings starts climbing, and by evening a chunk of “where’s my order?” questions get a polite non-answer.

You check the deploy log. Nothing shipped. Then you check the provider’s changelog: they rolled their hosted *-latest model to a new snapshot that morning. You didn’t deploy a change — they did, into your production system, with no canary and no gate, because from their side it was a routine upgrade and from your side it was invisible until users felt it.

Predict, before reading on: what one configuration choice would have turned that silent upstream swap into a change you got to test first?

Every change is a release

Classical deploy safety watches your commits. AI deploy safety has to watch a wider set of changes, because three things can alter your system’s behavior without a commit from you:

1. Your changes — a new prompt, a model swap, a retrieval tweak.
2. The provider’s changes — a hosted model updated, deprecated, or re-tuned; rate limits and defaults shifting under you.
3. The data’s changes — the help center gets reorganized and a third of your chunk index now points at stale sections (Chapter 0’s second failure).

Every one of these is a release: a change to the function users meet. The single highest-leverage habit is to pin versions so that (2) becomes opt-in. A pinned model id (claude-haiku-4-5, not *-latest) turns the provider’s surprise upgrade into a migration you schedule — one you can shadow, gate, and canary like your own code. You give up automatic access to improvements; you gain the right to test them before your users do. For anything user-facing, that trade is almost always correct.

The rollout ladder

Four mechanisms, each buying a distinct thing. They stack — a serious change uses all four — but you should be able to say what each one is for:

• Shadow — run the new version on a mirror of real traffic and log its outputs without showing them to anyone. Buys: a read on real-world behavior (and a golden-set score) at zero user risk. The only way to test a model change on your actual query distribution before committing to it.
• Canary — send a small slice (1% → 10% → 100%) to the new version, watching quality and cost at each step. Buys: a bounded blast radius. A regression hits 1% of users for ten minutes, not everyone for a day.
• Feature flag — a runtime switch for behavior (model, prompt, threshold) that flips without a redeploy. Buys: seconds-not-minutes rollback — a kill switch you pull while you debug the cause offline.
• Fallback — a defined safe behavior for when the primary path fails: a cheaper backup model, a cached answer, or an honest “I can’t answer that right now” instead of a timeout or a hallucination. Buys: graceful degradation instead of a hard outage.

In code, the flag and the fallback are a few lines — but they are the few lines that turn a 40-minute incident into a 10-second one:

def answer(question: str, context: str) -> str:
    """Model behind a flag (instant rollback), with a fallback path (graceful
    degradation). The flag value lives in config you can flip at runtime; the
    fallback is what users get when the primary path fails its checks."""
    model = flags.get("bot_model", "claude-haiku-4-5")   # pinned default, flip-able
    try:
        reply = call_model(model, question, context)
        if not passes_output_contract(reply):            # schema / guardrail check
            raise ValueError("primary reply failed validation")
        return reply
    except (TimeoutError, ValueError):
        return fallback_answer(question)                 # cached / cheaper / honest "I don't know"

Gate on quality, not on green

Chapter 0’s lesson was that infra dashboards watch the request, not the answer — so a deploy can be “healthy” (200s, fast) and still be worse. That means the promotion decision cannot be “the canary’s error rate is flat.” It has to be a quality gate: rerun the golden set against the new version and refuse to promote on a regression.

This is where deploy hands off to the Evaluation guide. The golden set — a few dozen labelled cases covering the long tail you fear — is the same artifact you built there; here it becomes the gate on every promotion, including the provider’s upgrade you shadowed. Shadow produces the new version’s answers; the golden set scores them; promotion is blocked if the score drops. No green dashboard substitutes for that rerun, because the dashboard cannot see the dimension that moved.

The shape to carry: shadow → score on the golden set → canary if it passes → flag so you can pull it → fallback for when you must. Every stage answers a different “what if,” and the gate in the middle is the one infra can’t fake.

See it: which safety step was skipped?

Three changes that reached production and broke. Predict the missing mechanism for each before you reveal it — then notice each maps to one rung of the ladder above.

The pattern: none of these were caught by a smoke test, and none would have been caught by a dashboard. Each needed a specific rung — a gate, a pin, a flag — and “we tested it” is not the name of any of them.

How this is graded

• Technical Correctness — this chapter’s weighted dimension: you name the right mechanism for the failure (a pin for an upstream swap, a flag for fast rollback, a gate for a quality regression) instead of saying “add monitoring” or “test more.”
• Trade-off Awareness — you weigh pinning (control vs missing improvements) and canary speed (safety vs velocity), and you know shadow costs real inference dollars.
• Evaluation Rigor — you make promotion gate on a golden-set rerun, not on infra health, because you remember the failure is silent.
• Communication — “a model swap is a release; we shadow it, gate on the golden set, canary behind a flag” is the sentence that says you’ve run one of these.

Industry variation

• Startups — the whole rollout might be a flag and a 30-case gate, flipped by hand. That’s enough; the unskippable rung is the gate, not the tooling.
• Enterprise — change control and audit trails make “who promoted what, against which gate score” a recorded artifact; shadow environments are standard.
• Regulated (fintech, health) — a model change can require re-validation and sign-off; pinning isn’t optional, and “the provider upgraded it” is not an accepted explanation for a behavior change.
• AI-assisted coding interviews — you may be asked to sketch the rollout for a model swap live; naming shadow/gate/canary/flag in order is the signal.

Stretch: the change was safe, but the system fell over anyway

You pinned the model, gated the swap, canaried behind a flag — the deploy was textbook. Then the marketing promo from Chapter 0 lands, traffic jumps 6×, and p99 latency blows through the SLA. Nothing about your change was unsafe; the system simply couldn’t serve the load. That isn’t a deploy problem at all — it’s the next stage of the loop. Where did the time actually go, and why does a traffic spike break latency when the per-request work never changed? (That’s Chapter 2 — latency anatomy, where a slow system turns out to be a queue, not a slow model.)