insideout Get Pro

12 · Distillation

Pro

Training and Tuning, lesson 12

Distillation

Train a small model on a big one's outputs.

Big models are good and costly to run. Distillation trains a small model on a big one's outputs, so you keep much of the quality at a fraction of the cost. You need softmax and cross-entropy from "Pretraining", and fine-tuning on examples from "Instruction tuning".

By the end, you can:

  1. Soften a model's probabilities with a temperature, and say what the soft version adds.
  2. Work out the KL loss between a teacher and a student, and scale it by T².
  3. Build a distillation set from a teacher's answers, filtered by a checker.
  4. Say why a student stays below its teacher, and when distillation beats training on human data.

Part 1. What a teacher's guesses hold

Soften a model's probabilities with a temperature, and say what the soft version adds.

Training text gives one right answer per position: after "The cat sat on the", it says "mat". A trained model says more. It might give "mat" most of its probability, "sofa" some and "moon" a sliver. That spread marks which mistakes are near misses, which one label can't do.

Example 1.1

Softening three guesses

A big model's logits for the next token after "The cat sat on the" are 4.0 for mat, 2.0 for sofa and 0.0 for moon. What are its probabilities at temperature T = 1, and at T = 4?

  1. Divide the logits by T(4.0, 2.0, 0.0) ÷ 4 = (1.0, 0.5, 0.0)

    At T = 1 they stay as they are.

  2. Softmax at T = 1(0.867, 0.117, 0.016)

    Moon gets under 2%, too little to learn much from.

  3. Softmax at T = 4(0.506, 0.307, 0.186)

    The same order, but the small chances now carry real weight.

  4. Compare mat with sofae² = 7.39 at T = 1, e^0.5 = 1.65 at T = 4

    Dividing by T shrinks every gap between logits by the same factor.

Answer(0.867, 0.117, 0.016) at T = 1, and (0.506, 0.307, 0.186) at T = 4.

Those small chances hold what the big model learned about how answers relate. Hinton, Vinyals and Dean's example: a picture of a BMW has a tiny chance of being taken for a garbage truck, but that's still far likelier than a carrot. A label just says "BMW".

chance of each next tokenT = 1T = 40.8670.506matlogit 4.00.1170.307sofalogit 2.00.0160.186moonlogit 0.0
chance of each next tokenT = 1T = 40.8670.506matlogit 4.00.1170.307sofalogit 2.00.0160.186moonlogit 0.0
Figure 1The same logits at two temperatures. At T = 1 moon all but disappears. At T = 4 every answer is visible, and mat still leads.

Teacher, student and soft targets

The big model is the teacher, and the small one being trained is the student. The teacher's probabilities are soft targets, softened by dividing each logit z by a temperature T before the softmax. T = 1 is the usual softmax, and higher T flattens it.

p_i = e^(z_i ÷ T) ÷ Σ e^(z_j ÷ T)

Temperature never changes which answer wins, only how sure the teacher sounds. Too high and everything looks alike: at T = 10 the chances are 0.402, 0.329 and 0.269. On digits, Hinton's team found any T above 8 suited a student with 300 units per layer, and 2.5 to 4 one with 30.

Your turn

A teacher's logits for three answers are 6, 3 and 0. What are its probabilities at T = 3?

Divide by 3 to get (2, 1, 0), then softmax: e² = 7.39, e¹ = 2.72 and e⁰ = 1 add to 11.11, giving (0.665, 0.245, 0.090). (0.950, 0.047, 0.002) is T = 1. (0.667, 0.333, 0) shares by the logits' sizes and skips the exponential. All equal would need an endless T.

Part 2. The distillation loss

Work out the KL loss between a teacher and a student, and scale it by T².

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. The distillation loss
  2. Learning from the teacher's answers
  3. The teacher sets the ceiling
See Pro plansI have Pro: sign in
Next lessonA fine-tuning run end to endData, training, evaluation and release for one fine-tuned model. Start lesson 13

Training and Tuning overviewSpotted a mistake? Tell us.