Bits per byte

Tokens and Embeddings, lesson 11

A loss you can compare across models with different tokenizers.

You know average negative log-likelihood ("A bigram language model") and that two tokenizers can turn one text into very different token counts ("Tokenizer quirks: numbers, spelling, languages").

Part 1. Loss per token depends on the tokenizer

Two teams train language models and test them on the same 1,000-byte passage. Team A's tokenizer uses one token per byte, so the passage is 1,000 tokens. Team B's is BPE, and its tokens cover 4 bytes on average, so the passage is 250 tokens.

Each team reports its average loss per token, the same kind of number as the bigram model's 1.631. A gets 1.2. B gets 3.6. (These results are made up, but they're the right shape.)

Your turn

Which model predicts the passage better?

A makes 1,000 predictions at 1.2 each: 1,200 in total. B makes 250 predictions at 3.6 each: 900 in total. Both totals are for the same passage, so B is less surprised by it. Each of B's predictions has to pin down 4 bytes at once, so each one costs more.

Show the answer

A. A makes 1,000 predictions at 1.2 each: 1,200 in total. B makes 250 predictions at 3.6 each: 900 in total. Both totals are for the same passage, so B is less surprised by it. Each of B's predictions has to pin down 4 bytes at once, so each one costs more.

Loss per token is an average over tokens, and the tokenizer decides how many tokens there are. The total loss over the passage doesn't depend on that, as long as both models read the same passage.

Part 2. From nats to bits

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

From nats to bits
Divide by the bytes
Why training runs report it

See Pro plans I have Pro: sign in

Next lessonHow embedding models are trainedTrain a model to rank each query's passage above the rest of its batch. Start lesson 12

Tokens and Embeddings overviewSpotted a mistake? Tell us.

11 · Bits per byte