Prompt engineering as a discipline

The prompt that worked once

You need to pull {name, amount, due_date} off incoming invoices. You write the obvious prompt — “Tell me the name, amount, and due date from this invoice” — and try it on three invoices from your inbox. Perfect every time. You wire it into the pipeline and ship.

The next morning the pipeline is full of errors. One invoice had two dates and the model returned a sentence. One was in euros and the amount came back as "€1,200". One had no due date and the model made one up. Nothing about the model changed — the prompt was never a spec, so the model improvised, and your parser had no idea.

Predict: name two things the original prompt failed to pin down.

A prompt is a spec

A reliable prompt pins down four things. Leave any of them unstated and the model picks for you — differently each call:

1. Role — who the model is acting as (sets vocabulary and defaults).
2. Context — the specific material to use, and the rule for when it’s missing.
3. Instruction — the task, and the guardrails (what not to do).
4. Output format — length, structure, and shape the rest of your system expects.

def build_prompt(question: str, context: str) -> str:
    """A prompt is a spec: role, context, instruction, and output format — each
    pinned so the model can't quietly decide it for you."""
    return (
        "You are a support assistant for ACME. "             # role
        "Answer ONLY from the context below; "               # instruction + guardrail
        "if the answer isn't there, say 'I don't know'.\n\n"  # the missing-context rule
        f"Context:\n{context}\n\n"                            # context
        f"Question: {question}\n\n"
        "Answer in at most 2 sentences, in plain language."  # output format
    )

Every clause is there to remove a degree of freedom the model would otherwise exercise on its own. “Answer ONLY from the context” is the difference between a grounded assistant and a confident fabricator; “if the answer isn’t there, say ‘I don’t know’” is the difference between a safe failure and a made-up one.

Make the output a contract

The invoice pipeline broke because code consumed prose. Anything a program reads needs a contract: a fixed structure you ask for and then validate. Don’t parse sentences — demand JSON and check it.

import json

SCHEMA = {"name": str, "amount": float, "due_date": str}

INSTRUCTION = (
    "Return ONLY a JSON object with keys: "
    "name (string), amount (number, no currency symbol), "
    "due_date (string, YYYY-MM-DD). Use null if a field is absent."
)

def parse_invoice(reply: str) -> dict:
    """Validate against the contract. On a miss you reject — or re-prompt to repair —
    instead of letting malformed data into the pipeline."""
    data = json.loads(reply)  # raises immediately if the model didn't return JSON
    for field, typ in SCHEMA.items():
        if field not in data:
            raise ValueError(f"missing field: {field}")
        if data[field] is not None and not isinstance(data[field], typ):
            raise ValueError(f"{field} should be {typ.__name__}, got {data[field]!r}")
    return data

Now the failures become visible and handleable: a non-JSON reply raises at json.loads, a missing due_date is null you can branch on (not an invented date), and "€1,200" fails the type check instead of poisoning a downstream sum. Structured output is what turns an LLM call into a reliable function call — the same idea behind “function calling” / “tool use” and library validators like Pydantic.

Prompts are code

The original prompt “worked on three examples.” That’s a demo, not a test. A prompt is code with an unusually stochastic runtime, so it earns the same discipline:

• Version it. Keep prompts in source control, not pasted in a hundred call sites. A prompt change is a code change with a diff and a reviewer.
• Keep a held-out test set. Ten to a few dozen labeled cases that cover the long tail you fear — empty input, two dates, another currency, a hostile user — not three from your inbox.
• Check for regressions on every edit. Re-run the set when you “improve” a prompt; a tweak that helps one case routinely breaks another. (How you score those outputs is the Evaluation guide — here, the habit is enough.)

The mindset shift: you are not crafting one perfect string, you are maintaining a small program against a distribution of inputs.

See it: which prompt is the engineering choice?

Three contrasts. Predict what actually makes the disciplined version robust before you reveal it — then notice each maps to one section above.

The pattern: robustness never came from nicer wording. It came from pinning the spec, demanding a contract, and testing against the distribution — the three habits that turn prompting from folklore into engineering.

How this is graded

• Technical Correctness — you distinguish prompt wording from prompt spec, and know structured output is a contract you validate, not a formatting preference.
• Trade-off Awareness — you know when a strict schema helps (machine-consumed output) and when it’s overkill (a human reads the reply), and you weigh re-prompting-to-repair against rejecting.
• Evaluation Rigor — this chapter’s weighted dimension: you refuse “worked on three examples” and reach for a held-out set and regression checks on every edit.
• Communication — “I’d return a validated JSON object with an enum for the label, and gate prompt changes on a held-out set” beats “I’d tweak the wording.”

Industry variation

• Startups — a versioned prompt + a 20-case eval set is often the entire eval infrastructure, and it’s enough to ship responsibly.
• Enterprise — prompts are reviewed artifacts with audit trails; structured output and validation are mandatory because the result crosses a system boundary.
• AI-assisted coding interviews — increasingly you’ll be asked to write and verify an LLM call live; the contract-and-test habit is exactly what’s being assessed.

Stretch: the prompt was never the bottleneck

You harden the prompt, add a schema, build a held-out set — and the assistant still gets refund questions wrong. The spec is perfect; the problem is that the right policy was never in the context window in the first place. No prompt can cite a document the model never saw. What layer is actually failing, and what would you build to fix it? (That’s retrieval — Chapter 2 — where getting the right context in front of the model becomes its own engineering problem.)