Masking future tokens

Transformers, lesson 4

A language model may only look backwards, and the mask enforces it.

You know scaled dot-product attention ("Why the scores are scaled"): scores, softmax, a blend of values. This lesson adds one rule about which scores are allowed.

Part 1. Every position is a guess

A language model is trained to guess the next token. Take the sentence "the cat sat on the mat". It holds more than one lesson: after "the" comes "cat", after "the cat" comes "sat", after "the cat sat" comes "on", and so on.

Attention handles every position at once, so one pass over the sentence can make all those guesses side by side and check every one of them. That's what makes training on huge piles of text practical.

Your turn

The training sentence is six tokens: the cat sat on the mat. How many next-token guesses can the model check in one pass?

Every position except the last has a known next token to check against: the → cat, cat → sat, sat → on, on → the, the → mat. The guess made at mat has no answer in this sentence, so it isn't scored. Five guesses from one pass.

Show the answer

A. Every position except the last has a known next token to check against: the → cat, cat → sat, sat → on, on → the, the → mat. The guess made at mat has no answer in this sentence, so it isn't scored. Five guesses from one pass.

Part 2. The leak

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

The leak
Minus infinity before softmax

See Pro plans I have Pro: sign in

Next lessonHeads in parallelSeveral smaller attention heads, each tracking a different relationship. Start lesson 5

Transformers overviewSpotted a mistake? Tell us.

04 · Masking future tokens