insideout Get Pro

05 · Overlap metrics: BLEU, ROUGE, BERTScore

Pro

Evaluation, lesson 5

Overlap metrics: BLEU, ROUGE, BERTScore

When matching a reference answer works, and when it misleads.

"Rubrics": people marking quality against a written guide. This lesson scores by counting words shared with a reference answer, which costs nothing and needs no grader.

Part 1. Counting shared words

Say a person wrote the ideal answer: the cat sat on the mat. Your model wrote: the cat is on the mat. They're close. To put a number on how close, count how many of the reference's words the model also used.

Example 1.1

Shared words, by hand

  1. referencethecatsatonthemat
    modelthecatisonthemat

    Six words each. Lowercase both and split on spaces.

  2. the ×2✓ 2
    cat✓on✓mat✓
    sat✗

    Go through the reference. "the" appears twice in both, so both count. cat, on and mat match. "sat" is missing.

  3. 5 matched÷6 in reference=0.83

    Five of the reference's six words turn up in the answer. That share is the score.

ROUGE

A family of scores from summarization research that measure how much of the reference shows up in the output. ROUGE-1 counts single words, as above. ROUGE-2 counts pairs of words, and ROUGE-L uses the longest run of words that appear in the same order. Libraries usually report recall, precision and F1 for each.

Your turn

Same reference. The model writes "the cat". What's its ROUGE-1 recall?

Recall asks how much of the reference you covered: 2 of 6 words, so 0.33. Option A describes precision (every word it wrote is in the reference), which is 1.0 here. That's why recall suits summaries: leaving things out costs you.

Part 2. Counting the other way

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Counting the other way
  2. When the counts lie
  3. Matching meaning instead of words
See Pro plansI have Pro: sign in
Next lessonModels as gradersUse one model to grade another, and check it against people. Start lesson 6

Evaluation overviewSpotted a mistake? Tell us.