insideout Get Pro

04 · Updating weights with gradients

Pro

Neural Networks, lesson 4

Updating weights with gradients

Move every weight against its gradient, then repeat until the loss falls.

"Loss functions": the loss is one number for how wrong a network is, and for the one-weight apple-price network it makes a bowl with its floor at w = 2.

Part 1. Guess, measure, nudge

Start with the oldest learning rule there is. Take a neuron like the one in "One neuron", with two inputs, two weights and a bias, but give it a hard yes-or-no activation: it says yes (1) if its total is zero or more, and no (0) otherwise.

At the start the weights are random, so its calls are random too. Learning means this: make a guess, see how wrong it was, and nudge the weights so the same input would land a bit closer next time. Then do it again, thousands of times.

Example 1.1

One nudge, by hand

  1. x₁ = 1x₂ = 0should sayyes (1)
    w₁ = 0.2w₂ = −0.5b = −0.4

    One example. The inputs are 1 and 0, and the right answer is yes. The weights start wherever they started.

  2. 1 × 0.2+0 × −0.5+−0.4=−0.2
    below zero, so it saysno (0)

    Multiply, add, add the bias. The total is −0.2, which is below zero, so the machine says no. It's wrong.

  3. error=1 − 0=1

    Measure how wrong: the right answer minus the guess. Here that's 1. If the guess had been right, the error would be 0 and nothing would change.

  4. w₁+0.5 × 1 × 1=0.7
    w₂+0.5 × 1 × 0=−0.5
    b+0.5 × 1=0.1

    Nudge each weight by step size × error × its input. The step size here is 0.5. w₂ doesn't move, because its input was 0: it had no part in the mistake.

  5. 1 × 0.7+0 × −0.5+0.1=0.8
    now it saysyes (1)

    Run the same input again. The total is 0.8, so it says yes. One nudge fixed this example.

Perceptron

This exact machine: weights, a bias and a hard yes/no cut. Frank Rosenblatt built one in 1958. The nudge you just did is the perceptron learning rule.

Your turn

The machine guesses yes, and the right answer is yes. What happens to the weights?

Every nudge is step size × error × input. A right guess has error 0, so the whole product is 0. The machine only changes when it gets something wrong.

Part 2. Follow the slope downhill

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Follow the slope downhill
  2. Every weight at once
See Pro plansI have Pro: sign in
Next lessonThe chain rule by handWork out each weight's gradient through a small network, step by step. Start lesson 5

Neural Networks overviewSpotted a mistake? Tell us.