insideout Get Pro

16 · Capstone: an evaluation report

Pro

Evaluation, lesson 16

Capstone: an evaluation report

Evaluate a real system and write up what you found.

This walkthrough uses the earlier lessons in this module, from "Specifying a good output" to "Multiple testing in experiments". Each step names the lesson it comes from.

Part 1. The system, and what good means for it

A support team hands tickets between shifts. Some threads run to 40 messages. A new system reads a thread and writes a handoff note of at most 60 words, so the next person can pick up without reading everything. Here's how the team evaluated it before switching it on.

First they read 80 notes from an early prototype and sorted what went wrong ("Read the failures first"). Two piles stood out: notes that claimed things the thread didn't say, and notes that left out what was still pending. Those became the first two criteria. Then, before the real run, they wrote the criteria and the decision rule down ("Specifying a good output"):

Example 1.1

Criteria, written in advance

  1. gatefaithful: every claim supported by the thread
    at least 95% overall, and at least 92% in every language

    A note that says "refund issued" when it wasn't is worse than no note. So faithfulness is a gate, checked per language as well as overall.

  2. outcomecomplete: covers the issue, what was tried, what's pending
    at least 85% of checklist items covered, on average

    A three-item yes or no checklist ("Rubrics"), because people agree more easily on yes or no.

  3. guardrailno card numbers or passwords copied into notes
    comparisonbeats the simple baseline, with an interval above 50%

    Plus a privacy guardrail checked by a pattern match on every note, and a requirement to beat the baseline clearly ("Baselines").

Your turn

Why did the team write "at least 92% in every language" as well as the overall bar?

"Read the failures first" showed how a minority slice can fail while the total looks fine. Writing the slice bar in advance means a bad slice blocks that part of the launch automatically, instead of being argued away later.

Part 2. A test set and a judge checked against people

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. A test set and a judge checked against people
  2. Against a baseline, with intervals
  3. Error analysis and the report
See Pro plansI have Pro: sign in
The last lesson in EvaluationBack to the overviewMeasuring what a system does Back to Evaluation

Evaluation overviewSpotted a mistake? Tell us.