Tokens and Embeddings, lesson 10
A bigram language model
Predict the next character by counting, then by training, and compare the two.
You know softmax and cross-entropy ("Backprop through softmax and cross-entropy") and that a lookup is a one-hot vector times a table ("Looking up a token's vector").
Part 1. Count what comes next
Here are 36 short first names, picked for this lesson: ada, ali, amir, ana, ari, bo, cleo, dara, eli, emil, ezra, hana, ida, ines, ivan, jada, kai, kira, lars, lea, leon, lina, luis, mara, mila, nadia, nils, nora, omar, rosa, sami, tara, theo, yara, zane, zoe.
The smallest model of names you can build looks at one letter and guesses the next. To learn it, you only need to count. Walk through each name a pair of letters at a time. Put a full stop before and after every name, so the model also learns how names start and end.
Your turn
Across all 36 names, what most often comes right after the letter "a"?
Example 1.1
Counting pairs
- mila→. mm ii ll aa .
One name gives five pairs, counting the start and the end. The full stop is a 27th symbol next to the 26 letters.
- after a:. 17r 7n 4d 3m 2i 1l 1
Tally every pair from every name. This is the row for "a": 35 pairs start with a, and 17 of them end the name.
- after .:a 5l 5e 3i 3n 3k 2m 2t 2z 2and 9 more letters, 1 each
The row for the full stop holds the first letters. Every name starts once, so this row adds up to 36.
- 27 rows × 27 columns170 pairs counted, most cells 0
The whole model is this table of counts: one row for each symbol, one column for each symbol that could follow it.
Bigram
Two neighbouring tokens, here two letters. A bigram model predicts the next token from the one before it, and nothing else.
Part 2. From counts to new names
Keep going with Pro
You've done the first part. Pro opens the rest of this lesson, and every other one.
- From counts to new names
- Score it on real names
- Smoothing away the zeros
- The same model, trained
Tokens and Embeddings overviewSpotted a mistake? Tell us.