insideout Get Pro

12 · Cutting the tail with top-k and top-p

Pro

Transformers, lesson 12

Cutting the tail with top-k and top-p

Sample only from the likeliest tokens to avoid rare nonsense.

You need "Turning logits into choices": dividing the logits by a temperature before softmax makes the odds sharper or flatter, but never removes a token.

Part 1. The tail adds up

After "The cat sat on the", a real model gives some probability to every one of its tens of thousands of tokens: "mat", but also "photosynthesis", "%" and half a word in Korean. Each junk token has a tiny chance. There are just so many of them.

Your turn

Say the junk tokens add up to 1% at each step. You sample a 100-token reply. How likely is at least one junk pick?

Each step avoids junk with chance 0.99, so all 100 steps avoid it with 0.99^100 ≈ 0.37. That leaves about 63% for at least one bad token. And one bad token in the context can drag the rest of the reply with it ("Autoregressive generation" showed why). The numbers here are illustrative, but the effect is real.

So the idea is to cut the bottom of the list off before rolling the dice, and share its probability among the tokens that survive. There are two common ways to decide where to cut.

Part 2. Top-k: keep a fixed number

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Top-k: keep a fixed number
  2. Top-p: keep enough to cover p
  3. Using them together
See Pro plansI have Pro: sign in
Next lessonSeeds, reproducibility and picking settingsGetting the same output twice, and which settings suit which task. Start lesson 13

Transformers overviewSpotted a mistake? Tell us.