insideout Get Pro

15 · Quantizing weights

Pro

Inference and Serving, lesson 15

Quantizing weights

Map weights to small integers with a scale and a zero point.

You know from "Number formats" that INT8 stores whole numbers from −128 to 127 and needs a scale to stand for real weights.

Part 1. A ruler with 256 marks

Here are six weights from one row of a layer: 0.423, −1.27, 0.0861, 0.951, −0.338 and 0.617. Each takes 4 bytes in FP32. You want to store each in 1 byte as INT8.

The plan: make the biggest weight (by size) land on the end of the integer range, and space everything else evenly between. Think of it as a ruler with 256 marks, stretched to fit the row.

Your turn

The biggest weight by size is −1.27. If −1.27 maps to −127, how much is one integer step worth?

1.27 ÷ 127 = 0.01. Every mark on the integer ruler is worth 0.01, so the integer 42 means 0.42 and −127 means −1.27.

Example 1.1

Quantizing the row to INT8 by hand

  1. largest |w|=1.27
    1.27÷127=scale 0.01

    Find the largest weight by size and divide by 127. That's the scale.

  2. 0.423−1.270.08610.951−0.3380.617
    ÷ 0.01
    42.3−1278.6195.1−33.861.7

    Divide every weight by the scale. Now they're on the integer ruler, but not on its marks yet.

  3. 42−127995−3462

    Round to the nearest whole number. These six bytes, plus one scale for the row, are all you store.

  4. × 0.01
    0.42−1.270.090.95−0.340.62

    To use a weight, multiply its integer by the scale. Close to the originals, but not equal.

Scale and symmetric quantization

The scale is the real value of one integer step. Choosing it from the largest absolute weight, so that 0 stays at integer 0 and the range reaches equally far both ways, is symmetric quantization. Multiplying an integer by the scale to get a real number back is dequantization.

Part 2. Every weight moves a little

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Every weight moves a little
  2. A zero point for lopsided ranges
  3. One scale, or many
See Pro plansI have Pro: sign in
Next lessonLayers that resist quantizationOutliers and sensitive layers, and how to find them. Start lesson 16

Inference and Serving overviewSpotted a mistake? Tell us.