Training and Tuning, lesson 5
Instruction tuning
Turning a base model into one that follows requests.
You know what a base model is from "Pretraining": a model that continues text rather than answering it, trained by scoring every next token.
Part 1. From continuing to answering
Type "What is the capital of France?" into a base model. It may answer. It may also write "What is the capital of Germany? What is the capital of Spain?", because lists of quiz questions are common on the web and carrying one on is a fine way to predict the next token.
The fix is more training, on a different pile: requests, each paired with a good answer. Some are written by people, some are written by a stronger model and checked. The loop is the same next-token loop as pretraining.
Your turn
Meta's Llama 2 chat model learned to answer from hand-written examples. About how many?
Quality matters more than count. A 2023 study called LIMA got a capable assistant out of a 65-billion-parameter base model with 1,000 carefully chosen examples.
Most instruction data today is written by stronger models and then filtered, with people writing and checking the hardest parts. Llama 3's tuning had its own models write several answers to each prompt and kept the best. Open mixes such as AI2's Tülu 3 and Hugging Face's SmolTalk show what such a set looks like.
Instruction tuning (SFT)
Training a base model on request and answer pairs so it follows requests. It's also called supervised fine-tuning, or SFT. The result is an instruct model, or chat model.
Part 2. Chat templates
Keep going with Pro
You've done the first part. Pro opens the rest of this lesson, and every other one.
- Chat templates
- Score only the answer
- Gentle training
Training and Tuning overviewSpotted a mistake? Tell us.