Grammars that constrain decoding

Prompting, lesson 5

Mask invalid tokens at each step so output always fits a grammar.

You know a model gives a score to every token in its vocabulary at each step ("Logits and softmax over the vocabulary" in Transformers), and why apps want replies that fit a schema ("JSON that always parses").

Part 1. Some tokens would break the format

A model is tagging product reviews. The schema says sentiment must be one of three exact words: positive, negative or neutral. The review is "Love it, works great." and the model has written {"sentiment": " so far.

Here are its probabilities for the next token. To keep the sums small, pretend the vocabulary has only five tokens: positive 40.1%, Positive 26.9%, mixed 18.0%, negative 9.0%, neutral 6.0%.

Your turn

If you sample normally, how likely is this step to break the schema?

"Positive" with a capital P (26.9%) and "mixed" (18.0%) both break a schema that allows three exact lowercase words. Together that's 44.9%. The model has the right idea, but nearly half its weight sits on spellings your code will reject.

Show the answer

D. "Positive" with a capital P (26.9%) and "mixed" (18.0%) both break a schema that allows three exact lowercase words. Together that's 44.9%. The model has the right idea, but nearly half its weight sits on spellings your code will reject.

Part 2. Mask, then sample

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Mask, then sample
A grammar decides the mask
Valid isn't the same as right

See Pro plans I have Pro: sign in

Next lessonLetting the model call functionsTool definitions, tool calls, and the loop that runs them. Start lesson 6

Prompting overviewSpotted a mistake? Tell us.

05 · Grammars that constrain decoding