insideout Get Pro

06 · Autograd you can step through

Pro

Neural Networks, lesson 6

Autograd you can step through

Write backward() for a few operations and watch gradients flow through a graph.

"The chain rule by hand": a backward pass multiplies local rates from the loss back to each weight, and nudging checks the result.

Part 1. A number that remembers

Doing the backward pass by hand worked for four parameters. For millions, the bookkeeping has to happen on its own. The trick is to make every number remember how it was made.

Wrap each number in a small object. When two of them are multiplied, the result keeps a note: "I am a times b". When that's added to c, the next result notes "I am that, plus c". By the end, the output holds a record of every step that led to it.

Example 1.1

Building out = tanh(a·b + c)

  1. a = 2.0b = −0.4c = 1.6

    Three plain inputs. Nothing made them, so they have no record.

  2. p = a × b=−0.8
    remembers×, from a and b

    p keeps its value and a note of what made it.

  3. q = p + c=0.8
    remembers+, from p and c

    q points back to p, which points back to a and b.

  4. out = tanh(q)=0.664
    rememberstanh, from q

    Following the notes back from out reaches every input.

Your turn

Python runs q = p + c, with p = −0.8 and c = 1.6. Besides the value 0.8, what does q need to keep for the backward pass later?

Each value only needs its own parents and the rule that joined them. Chain those small records together and you can walk from the output back to every input, without any one value knowing the whole formula.

Computation graph

The record of how a result was made. Each value is a node, linked to the values it was made from. Following the links back from the output reaches every input.

Part 2. Each operation knows its rule

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Each operation knows its rule
  2. The right order
  3. Used twice, gradients add
See Pro plansI have Pro: sign in
Next lessonBackprop through softmax and cross-entropyThe gradient every classifier and language model starts from, derived by hand. Start lesson 7

Neural Networks overviewSpotted a mistake? Tell us.