insideout Get Pro

10 · Autoregressive generation

Pro

Transformers, lesson 10

Autoregressive generation

Pick a token, append it, run the model again.

You need "Logits and softmax over the vocabulary": a forward pass ends with a probability for each token the model knows.

Part 1. One token, then the next

A forward pass gives you one set of probabilities for the next token. That's one token. To write a sentence, you take a token, glue it onto the end of the text, and run the model again on the longer text.

Example 1.1

Writing after "The cat sat on the"

  1. The cat sat on the
    →forward pass→mat 0.41floor 0.20...

    Run the model on the prompt. The last position gives probabilities for every token.

  2. pick→mat

    Choose one token from that list. How to choose is a separate decision, covered over the next few lessons.

  3. The cat sat on the mat
    →forward pass→. 0.52and 0.21...

    Append it and run the model again. The new token is now part of the input, just like the words you typed.

  4. pick→.
    The cat sat on the mat.

    Pick, append, repeat. Every token in a reply comes out of this same loop. (The probabilities after "mat" are illustrative.)

Autoregressive generation

Producing text one token at a time, where each new token is chosen from a forward pass over everything so far, including the tokens the model itself just wrote.

Your turn

A chatbot writes a 40-token reply to your message. How many times did it pick a token?

Each forward pass gives probabilities for one next token, so 40 tokens need 40 picks, each after its own pass. That's why replies stream out piece by piece, and why long replies take longer. "Reading the prompt, writing the reply", in Inference and Serving, looks at how servers make those passes cheaper.

Part 2. Knowing when to stop

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Knowing when to stop
  2. No going back
See Pro plansI have Pro: sign in
Next lessonTurning logits into choicesGreedy picks, and how temperature sharpens or flattens the odds. Start lesson 11

Transformers overviewSpotted a mistake? Tell us.