People as graders

Evaluation, lesson 8

When to use human raters, and how far to trust their agreement.

"How model graders are biased": checking a model judge against human labels. This lesson is about getting those labels, and knowing when they can be trusted.

Part 1. When people are worth it

People are slow and expensive graders, and they get tired. A model judge is fast and cheap. Yet every judge in this module was checked against people, because only people can say what "good" means for your users in the first place.

Your turn

A team has a calibrated judge running on every change. Where do human graders still earn their cost?

People set the standard the judge copies, so they're needed to build and recheck the labelled sample. They're also needed where a judge can't be trusted: medical or legal accuracy, brand voice, a new task with no rubric yet. Re-grading everything throws away the judge's savings.

Show the answer

C. People set the standard the judge copies, so they're needed to build and recheck the labelled sample. They're also needed where a judge can't be trusted: medical or legal accuracy, brand voice, a new task with no rubric yet. Re-grading everything throws away the judge's savings.

Use people to set the standard, to check the automatic graders against it from time to time, and to make the final call on a big launch. Use automatic graders for everything in between.

Part 2. Do the graders agree with each other?

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Do the graders agree with each other?
High agreement, no information
Planning a human study

See Pro plans I have Pro: sign in

Next lessonBaselinesCompare against the simplest thing that could work. Start lesson 9

Evaluation overviewSpotted a mistake? Tell us.

08 · People as graders