Treating prompts like code

Evaluation, lesson 13

Version prompts and run regression tests before every change ships.

"Specifying a good output" and "Models as graders": criteria you can check, and a model that checks the ones code can't. The Prompting module covers writing prompts; this lesson is about changing them safely once they're live.

Part 1. An edit nobody can trace

A support bot's system prompt lives in an admin panel. On Monday someone adds "Keep answers under 50 words." On Wednesday someone else fixes a typo and adds a line about refunds. On Friday complaints about refund answers double.

Your turn

Which edit caused Friday's complaints?

Both stories sound plausible, and that's the trap. Without a record of which prompt produced which answer, you can only guess, and you can't go back to Monday's version because nobody kept it.

Show the answer

A. Both stories sound plausible, and that's the trap. Without a record of which prompt produced which answer, you can only guess, and you can't go back to Monday's version because nobody kept it.

A prompt decides what your product does, just as code does. So give it what code gets: a file in the repository, a history of changes, tests before it ships, and a way back. The file holds the text plus everything else that shapes the answer.

A versioned prompt, in Python

import hashlib
from string import Template

PROMPT = {
    "id": "triage",
    "version": 4,
    "model": "example-model-2025-06-01",
    "temperature": 0,
    "template": "You sort support emails into one of: payment, delivery, other.\n"
                "Reply with the label only.\n\nEmail:\n$email\n",
}

text = PROMPT["template"]
fingerprint = hashlib.sha256(text.encode()).hexdigest()[:8]
filled = Template(text).substitute(email="I was charged twice.")

print(PROMPT["id"], "v" + str(PROMPT["version"]), fingerprint)
print(filled.splitlines()[-1])
# triage v4 0240299f
# I was charged twice.

Three habits come with the file. Every change is a commit with a reason ("v4: ask for the label only; v3 added explanations"). The prompt id and version are logged with every response the app sends. And the fingerprint, a hash of the exact text, catches the edit where someone changed the words but forgot to bump the number.

Prompt version

One fixed copy of a prompt and its settings, with a label you can point to. When an answer looks wrong, the version in the log tells you exactly which text produced it.

Your turn

A user reports a bad answer from last Tuesday. Which logged data lets you reproduce it?

Reproducing an answer means feeding the same input to the same prompt, model and settings. The current file may have changed three times since Tuesday. With the version logged, you check that one out and run it again.

Show the answer

A. Reproducing an answer means feeding the same input to the same prompt, model and settings. The current file may have changed three times since Tuesday. With the version logged, you check that one out and run it again.

Part 2. Pin everything, ship by pointer

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Pin everything, ship by pointer
A fix that breaks something else
Judged checks and repeated runs

See Pro plans I have Pro: sign in

Next lessonTesting a prompt changeMeasure a change on the task itself, not on how the output reads. Start lesson 14

Evaluation overviewSpotted a mistake? Tell us.

13 · Treating prompts like code