Blog9 min read
How large language models work
From your text to the next word, with every number shown.
The short answer
A large language model reads text as tokens, small pieces of words, and turns each one into a list of numbers. Layers of attention let every token take in what came before it. At the end, the model scores every token it knows, turns the scores into probabilities and picks one. Then it does the whole thing again for the next token. Training is how the numbers inside get set: the model guesses the next token across trillions of examples and is nudged toward the right answer each time.
Text becomes tokens
Why a model never sees letters, only numbers.
A model can't read letters. Before anything else, a tokenizer cuts your text into pieces called tokens and swaps each piece for a number, its ID. Common words get a token of their own. Rarer words are built from smaller pieces.
Token
A piece of text the model treats as one unit: a whole word, part of a word, a space plus a word, or a single character. Each token has an integer ID. With GPT-style tokenizers, 100 tokens is roughly 75 English words.
| Text | Tokens in GPT-2 | Token IDs |
|---|---|---|
| strawberry, at the start of a text | st · raw · berry | 301, 1831, 8396 |
| ␣strawberry, inside a sentence | ␣strawberry | 41236 |
| unbelievable | un · bel · iev · able | 403, 6667, 11203, 540 |
| ␣12345 | ␣123 · 45 | 17031, 2231 |
GPT-2's tokenizer has 50,257 tokens in its vocabulary. Newer models use bigger vocabularies, which fit the same text into fewer tokens. The pieces are chosen by byte pair encoding: start from single bytes, and keep merging the pair that appears most often in a big pile of text until the vocabulary is full. Training a BPE tokenizer runs the merges by hand.
This is why some chat models miscount the r's in "strawberry". In GPT-2, inside a sentence, the model receives one number, 41236, not ten letters. Newer tokenizers also make it a single token, with a different ID. It has to have learned how that token is spelled, and the count is easy to get wrong. Tokenizer quirks works through it.
Tokens become vectors
Each ID picks a row from a table of learned numbers.
Next, each token ID picks a row from a big table called the embedding table. The row is a vector: a list of numbers, 768 of them in the smallest GPT-2. The table has one row per token, so GPT-2's holds 50,257 × 768 = 38,597,376 numbers before the model has done anything else.
Embedding
The vector a token ID looks up. The numbers start random and are learned in training, so tokens used in similar ways end up with similar vectors.
A vector on its own doesn't say where the token sits in the text, and order matters: "dog bites man" isn't "man bites dog". So the model also gets told where each token sits. GPT-2 adds a learned vector for each position. Many newer models instead turn the query and key vectors used in attention, below, by an angle that depends on the position.
Attention: each token looks back
How a word picks up meaning from the words around it.
A word's meaning depends on its neighbours. "Bank" means one thing next to "river" and another next to "loan". Attention is how the model lets each token take in information from the tokens before it.
Three learned tables turn each token's vector into three new ones: a query, what this token is looking for; a key, what it offers to others; and a value, what it passes on. A token compares its query with every earlier key. A high score means that token matters here, and the token takes a weighted average of the values.
Example 3.1
Attention for one token, by hand
Three tokens have keys k1 = (1, 0), k2 = (0, 1) and k3 = (1, 1), and values v1 = (1, 0), v2 = (0, 2) and v3 = (3, 3). The third token's query is q3 = (1, 1). The vectors have 2 numbers, so scores are divided by √2 ≈ 1.414. What does the third token take in?
- Score each key against the queryq3·k1 = 1, q3·k2 = 1, q3·k3 = 2
A dot product: multiply matching numbers and add them. A bigger score means a better match.
- Scale the scores1 ÷ 1.414 = 0.71, 0.71, 2 ÷ 1.414 = 1.41
Dividing by the square root of the vector size stops scores from growing with the size and pushing softmax to extremes.
- Raise e to each scoree^0.71 = 2.03, 2.03, e^1.41 = 4.11; total 8.17
This is softmax: every result is positive, and dividing by the total makes them add to 1.
- Divide by the total0.25, 0.25, 0.50
The third token gives half its attention to itself and a quarter to each of the others.
- Mix the values0.25 × (1, 0) + 0.25 × (0, 2) + 0.50 × (3, 3) ≈ (1.76, 2.01)
Worked with the unrounded weights 0.248, 0.248 and 0.504. This new vector is what the token carries forward.
AnswerAbout (1.76, 2.01): mostly its own value, with some of each earlier token mixed in.
A real model does this with vectors of hundreds or thousands of numbers, in many attention heads side by side, each free to look for something different. Then a small feed-forward network works on each token's vector on its own. Attention plus that network makes one transformer block, and models stack many blocks: the smallest GPT-2 has 12, and GPT-3 has 96. Attention as a soft lookup builds it step by step.
Causal mask
A rule that stops a token from attending to the tokens after it. In training the model sees whole passages at once, and without the mask it could peek at the very word it is meant to predict.
From the last vector to the next word
Scores for every token, turned into probabilities, then one pick.
After the last block, the final token's vector is compared with every token in the vocabulary. That gives one score per token, called a logit: 50,257 of them for GPT-2. Softmax turns the scores into probabilities that add up to 1.
Example 4.1
Three candidates for the next word
After "The cat sat on the", suppose only three tokens have real scores: ␣mat 2.0, ␣sofa 1.0 and ␣roof 0.1. What are their probabilities?
- Raise e to each scoree^2.0 = 7.39, e^1.0 = 2.72, e^0.1 = 1.11
Every score becomes positive, and the gaps between them stretch.
- Add them up7.39 + 2.72 + 1.11 = 11.21
- Divide each by the total7.39 ÷ 11.21 = 0.66, 2.72 ÷ 11.21 = 0.24, 1.11 ÷ 11.21 = 0.10
Now they add up to 1.
Answer␣mat 66%, ␣sofa 24%, ␣roof 10%.
Then the model picks one. Always taking the top token gives flat, repetitive text, so most apps sample: they draw a token at random using those probabilities. Temperature sets how adventurous the draw is. Every score is divided by the temperature before softmax: at 0.5 the same three scores give 86%, 12% and 2%, and at 2.0 they give 50%, 30% and 19%.
Temperature
A number the scores are divided by before softmax. Below 1 it sharpens the probabilities toward the top token. Above 1 it flattens them. At 0, the model always takes the top token.
One token at a time
Why replies stream in a word or two at a time.
The chosen token is added to the end of the text, and the whole process runs again to pick the next one. That's why chat replies appear a word or two at a time: every new token needs its own trip through every block of the model.
A reply has two phases. First, prefill reads your whole prompt in one go. Then decode writes the answer one token at a time. To avoid redoing work, the model keeps every earlier token's keys and values in memory, the KV cache, so each new token computes only its own. That cache is a big part of why long conversations cost more memory and money. Reading the prompt, writing the reply shows both phases with real numbers.
Context window
The most tokens a model can take in at once: your prompt, the conversation so far and the reply it is writing. Anything beyond it has to be cut or summarised before the model sees it.
How the numbers get set
Training is guessing the next token, again and again.
Every number in the model starts random: the embedding table, the query, key and value tables, the feed-forward networks. Training sets them. The recipe is simple to say: show the model real text, ask it to predict each next token, and measure how wrong it was.
Example 6.1
Scoring one guess
The true next word was ␣mat, and the model gave it 66%. What is the loss? What would it have been if the model had given ␣mat only 10%?
- Loss at 66%−ln(0.66) = 0.42
The loss is minus the log of the probability given to the right token. This is called cross-entropy.
- Loss at 10%−ln(0.10) = 2.30
Being confidently wrong costs far more than being nearly right.
Answer0.42 at 66%, and 2.30 at 10%: more than five times worse.
Calculus then tells every number which way to move to make that loss a little smaller, and by how much. Working that out is backpropagation, and taking the step is gradient descent. Repeat it over a huge pile of text and the model gets very good at predicting the next token. GPT-3, with 175 billion numbers, was trained on about 300 billion tokens. Meta's Llama 3 8B and 70B were trained on over 15 trillion tokens.
A model trained only this way continues text. It doesn't follow instructions. To make an assistant, it's trained again on examples of good conversations, then tuned on feedback about which answers people prefer. Pretraining covers the first stage.
What this explains
Common model quirks, traced back to the mechanics.
- Miscounted letters. The model sees token IDs, not letters.
- Confident wrong answers. The model writes text that is likely, not text that has been checked. If a wrong answer looks likely, it comes out sounding as sure as a right one.
- Different answers to the same prompt. Sampling draws at random, so two runs can differ.
- Long prompts cost more. Every token is processed and kept in the KV cache, and most providers charge for tokens in and out.
- Out-of-date facts. The model knows what was in its training text. Anything newer has to go into the prompt, which is what retrieval does. See RAG vs fine-tuning.
Each of these has a fix you can build once you can see the mechanics. That's the idea behind insideout's AI lessons: watch every step happen with small numbers, then build it yourself, starting with one neuron.
Learn it properly
- Text as bytes, bytes as tokensTokens and Embeddings · Lesson 01 Free
- Tokenizer quirks: numbers, spelling, languagesTokens and Embeddings · Lesson 03 Free
- Context changes meaningTransformers · Lesson 01 Free
- Attention as a soft lookupTransformers · Lesson 02 Free
- Why the scores are scaledTransformers · Lesson 03 Free
- Reading the prompt, writing the replyInference and Serving · Lesson 01 Free
- PretrainingTraining and Tuning · Lesson 02 Free
- One neuronNeural Networks · Lesson 01 Free
Questions
What is a large language model, in one sentence?
A program that predicts the next token of a text, using billions of numbers learned from a huge amount of writing, and runs again and again to write a reply.
What is the difference between a token and a word?
A token is the unit the model reads. Common words are one token, often including the space before them, and rarer words split into several. With GPT-style tokenizers, 100 tokens is roughly 75 English words.
What does temperature do?
It divides the model's scores before they become probabilities. Low temperature makes the top token more likely and the text more predictable. High temperature spreads the probability out and the text gets more varied.
Why do models have a context window?
Attention compares each token with every token before it, and the model keeps keys and values for every token it has seen. Both grow with length, so memory and compute set a ceiling, and each model is trained up to a certain length.
Does a language model check its facts?
Not by itself. It writes what is likely given its training and your prompt. Apps add retrieval to supply sources and evaluation to catch mistakes. Evaluation shows how to measure them.