insideout Get Pro

14 · Testing a prompt change

Pro

Evaluation, lesson 14

Testing a prompt change

Measure a change on the task itself, not on how the output reads.

"Treating prompts like code" and "Is the new version really better": versioned prompts with regression tests, and a paired test for two versions. This lesson decides whether a new prompt is actually better.

Part 1. Similar text is not a finished task

A customer asks when their refund will arrive. The reference answer says: "Your refund will arrive within 5 to 7 working days." Two prompt versions reply:

A: "Your refund will arrive within 3 to 5 working days." B: "You'll see the money back in 5 to 7 working days."

Your turn

A scoring script counts how many words each reply shares with the reference. Which reply scores higher, and which is right?

A copies almost every word of the reference and changes the one fact that matters. B rewords everything and keeps the fact. Word-overlap scores reward A. The customer would be misled by A and helped by B.

So measure what the reply was for. For a refund question, that's "is the date right?". For triage, "did the email reach the right team?". For a support bot, "was the issue resolved without a human?".

Task completion

Whether the output did the job it was meant to do, checked by the outcome rather than the wording. It's harder to measure than text similarity, and much closer to what users care about.

Part 2. Offline: compare on the same cases

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Offline: compare on the same cases
  2. Online: an A/B test with real users
  3. Count the cost of the whole task
See Pro plansI have Pro: sign in
Next lessonScoring generated textFluency, faithfulness and relevance, and which metric fits which. Start lesson 15

Evaluation overviewSpotted a mistake? Tell us.