Diagnosing a network that won't train

Neural Networks, lesson 8

Dead units, exploding gradients and bad starting weights, and the fixes.

"Updating weights with gradients" (the learning rate) and "The chain rule by hand" (how a gradient passes back through tanh).

Part 1. Read the loss first

When training goes wrong, the loss usually tells you how before anything else does. Print it every so often and look at its shape.

Here is the apple-price network from "Loss functions" again, started at w = 1 and trained with three different learning rates.

Example 1.1

One network, three learning rates

rate 0.001
4.667→4.580→4.495→4.411
Too low. The loss falls, but so slowly it looks flat. Each step covers less than 1% of the way to the floor.
rate 0.25
4.667→8.296→14.749→26.220
Too high. Each step overshoots the floor by more than the last, and the loss grows. In a big network this ends with the loss printed as nan.
rate 0.075
4.667→0.420→0.038→0.003
About right. Big steps while the slope is steep, small ones near the floor.

NaN

Short for "not a number": what floating-point maths gives for things like 0 ÷ 0 or infinity minus infinity. Anything computed from a NaN is NaN too, so one bad value spreads through a whole network within a step or two.

Your turn

A loss reads 2.31, 2.05, 9.7, 3,400, then nan. What should you try first?

A loss that falls and then shoots up is the classic sign of steps that are too big, like the rate of 0.25 above. Cut the learning rate (3 to 10 times smaller is a common first try) and run again.

Show the answer

B. A loss that falls and then shoots up is the classic sign of steps that are too big, like the rate of 0.25 above. Cut the learning rate (3 to 10 times smaller is a common first try) and run again.

In practice

Large training runs log two numbers at every step: the loss, and the gradient norm, the overall size of the whole gradient with every weight counted together. Most also clip it: if the norm passes a limit, the whole gradient is scaled down to that limit before the update. Llama 2 was trained with a limit of 1.0, and with a learning rate that warmed up from near zero over the first 2,000 steps, so the earliest steps couldn't throw the weights around.

A loss that doesn't move at all, from the very first step, is a different problem. Often the gradient isn't reaching the weights. The next two sections cover the usual reasons.

Part 2. Neurons that stop answering

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Neurons that stop answering
Gradients that shrink or grow with depth
Check before a long run

See Pro plans I have Pro: sign in

Next lessonMomentum, Adam and learning-rate schedulesHow real training runs set the size of every weight's step. Start lesson 9

Neural Networks overviewSpotted a mistake? Tell us.

08 · Diagnosing a network that won't train