Incidents & reliability ops — SLOs, runbooks, degradation modes

The bad day arrives

You pinned versions, gated deploys, instrumented the loop, and built guardrails. Then, on an ordinary Tuesday, the model provider has a regional outage and every request starts timing out; or a prompt change slips the gate and refusal rate doubles; or a novel attack gets past the input classifier. Production’s defining property is that the failure you didn’t anticipate eventually happens, and on that day prevention is no longer the question — bounding the damage is.

Reliability ops is the discipline of the bad day: deciding in advance how reliable is reliable enough, how the system fails when it fails, and what the human on-call actually does at 3 a.m. The interview tell is whether a candidate treats reliability as a number to manage or as a vague aspiration to “be more careful.”

Predict, before reading on: leadership says the reliability target is “zero incidents.” Why is that not an engineering target — and what number would you propose instead?

SLOs and error budgets

A Service Level Objective is the target: “99.9% of requests succeed,” “p99 < 2s,” or — for an AI system — “groundedness ≥ 0.70 on sampled traffic.” Its shadow is the error budget: 99.9% good means 0.1% allowed-bad, and that allowance is a resource. If you’re under budget, you can ship faster and take more risk; if you’ve burned it, you freeze features and spend on stability. The budget converts a vague “be reliable” into an explicit dial between velocity and safety that the whole team can see.

The AI-specific move is that the SLO covers quality, not just uptime. A bot can be 100% available and 60% grounded — the classic three pillars (Chapter 5) would call it healthy. So the reliability targets that matter here are written against the fourth pillar: groundedness, refusal rate, escalation precision. “Is it up?” and “is it right?” each get their own SLO and their own budget.

Degradation modes and runbooks

Two controls turn a failure into a managed event:

• A degradation mode is the planned answer to “what happens when the primary path fails?” It’s Chapter 1’s fallback, promoted to a reliability control: on a provider outage, drop to a cached answer, a cheaper backup model, or an honest “I can’t answer right now” — graceful degradation instead of a wall of 500s. (And never a tight retry loop against a struggling provider — that’s a thundering herd that makes the outage worse.)
• A runbook is the tested procedure for a known alert: what it means, how to mitigate now, and who to escalate to. It exists so that the person on-call at 3 a.m. — who may have never seen this failure — executes a checklist instead of improvising. The first step is almost always the kill switch or flag from Chapter 1: mitigate, then debug.

RUNBOOK — quality alert: groundedness SLO breached
  1. MITIGATE  flip cascade flag → frontier-only (kills a miscalibrated-cheap regression)
               if still breaching, roll back to last known-good release
  2. ASSESS    check recent changes: deploy? corpus re-index? provider model update?
  3. ESCALATE  page the on-call ML engineer if mitigation doesn't recover within 15 min
  4. ROOT-CAUSE  (after mitigation) bisect the change; add a regression case to the gate

Mitigate, then fix

The single most important ordering under stress: stop the bleeding before you find the cause. Mitigation (flip a flag, roll back, fall back to the backup model) bounds the damage in seconds and buys you the time to do root-cause analysis calmly. The failure mode is inverting them — debugging the interesting question of why groundedness dropped while users keep getting bad answers, because the fix feels more satisfying than the rollback.

Root-cause still matters: a mitigation you never explain is a time bomb, and the durable fix (a new regression case in the gate, a corpus re-index, a guardrail for the new attack) is what stops the repeat. But it comes second. The runbook encodes the order so that an exhausted human at 3 a.m. doesn’t have to rediscover it.

See it: the bad day, three moments

Three moments from an incident, each with a reliability tool that fits. Predict the tool before the reveal — SLO/budget, degradation mode, or runbook.

The pattern: reliability isn’t heroics, it’s preparation. An error budget decided the risk tolerance in advance, a degradation mode decided how to fail in advance, and a runbook decided the response in advance — so the bad day is execution, not invention.

How this is graded

• Technical Correctness — you define an SLO and error budget correctly, know it covers quality not just uptime, and name degradation modes and runbooks as the concrete controls.
• Trade-off Awareness — this chapter’s weighted dimension: you treat the error budget as the explicit velocity-vs-reliability dial, and weigh degradation options (cost, staleness) against a hard failure.
• Evaluation Rigor — your SLO is a measured claim on sampled live traffic (Chapter 6), and your runbook adds a regression case so the incident becomes a permanent test.
• Communication — “we mitigated by flipping to frontier-only, then root-caused a corpus re-index, and added a gate case” is the incident narrative that signals seniority.

Industry variation

• Startups — one SLO, a flag-based kill switch, and a one-page runbook is a real reliability program; the senior move is having a degradation mode at all.
• Enterprise — formal SLOs per service and tenant, error-budget policy that gates feature work, on-call rotations, and blameless postmortems.
• Regulated — incidents carry disclosure and audit duties; runbooks and postmortems are controlled documents, and “we mitigated then investigated” must be evidenced.
• AI-assisted coding interviews — asked “what happens when it breaks,” the signal is mitigate-then-fix and a concrete degradation mode, not “we’d look at the logs.”

Stretch: every chapter assumed an API — should it have?

You’ve now run the entire production loop — deploy, serve, observe, evaluate, respond — and almost every chapter quietly assumed you were calling a hosted model API. But under all of it sits a standing architectural decision you never made explicit: self-host the model, or call an API? It changes your latency, your cost curve, your data exposure, and which of the last nine chapters become your operational problem versus the provider’s. When does the answer flip — and is cost even the thing that decides it? (That’s Chapter 10 — self-host vs API vs hybrid.)