insideout Get Pro

08 · Adapters and LoRA

Pro

Training and Tuning, lesson 8

Adapters and LoRA

Train a small add-on instead of the whole model.

You know fine-tuning from "Instruction tuning": more training on a smaller, targeted dataset. It helps to know that a layer's weights are a grid of numbers ("Stacking neurons into layers" in Neural Networks).

Part 1. What full fine-tuning costs

Fine-tuning every weight of a 7-billion-parameter model takes far more memory than running it. Each parameter brings luggage.

Example 1.1

Memory for full fine-tuning, 7B

  1. weights7B × 2 bytes=14 GB

    The weights in a 16-bit format. This is all you need to run the model.

  2. gradients7B × 2 bytes=14 GB

    One gradient per weight, from the backward pass.

  3. Adam7B × 12 bytes=84 GB

    The Adam optimiser keeps a 32-bit copy of every weight plus two running averages per weight: 4 + 4 + 4 bytes.

  4. total7B × 16 bytes=112 GB

    And that's before activations. It's more than one 80 GB GPU holds, for a model that runs in 14 GB.

Your turn

You want 20 versions of a 7B model, each tuned for a different customer. With full fine-tuning, what do you store?

Full fine-tuning changes every weight, so each version is a whole new model. 20 × 14 GB = 280 GB, and serving them means loading each one separately.

Full fine-tuning

Training every parameter of a pretrained model. It's the most flexible option and the most expensive one.

Part 2. A small add-on beside the big matrix

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. A small add-on beside the big matrix
  2. Starting harmless, ending free
  3. QLoRA: shrink the frozen part
See Pro plansI have Pro: sign in
Next lessonPrompt, retrieve or fine-tuneHow to choose between the three ways to adapt a model to your task. Start lesson 9

Training and Tuning overviewSpotted a mistake? Tell us.