Neural Networks, lesson 3
Loss functions
One number for how far off the network is, and why its shape matters.
A network with random weights gives bad answers, and before it can get better it needs to know how bad. This lesson turns "how wrong" into one number, in two ways, and draws how that number moves as a weight moves. You need the forward pass from the last lesson and a calculator with an ln key.
By the end, you can:
Part 1. One number for how wrong
Score a set of predictions with mean squared error.
Before a network can improve, it needs a measure of how bad its answers are. Not a feeling: one number, where smaller means better. Say a network forecasts tomorrow's top temperature, and you have three days of its forecasts next to what really happened.
Example 1.1
Three forecasts
The forecasts were 21, 17 and 30 °C. The real temperatures were 18, 20 and 30 °C. Score the forecaster with one number.
- Find each error21 − 18 = 3, 17 − 20 = −3, 30 − 30 = 0
Forecast minus what happened.
- Try adding them3 + (−3) + 0 = 0
That looks perfect, because misses in opposite directions cancel.
- Square each error3² = 9, (−3)² = 9, 0² = 0
Now a miss counts whichever way it went, and every square is 0 or more.
- Average the squares(9 + 9 + 0) ÷ 3 = 6
A perfect forecaster would score 0.
AnswerA mean squared error of 6.
Squaring fixes the cancelling, and it does one more thing: a big miss now costs far more than a small one. A miss of 2 costs 4. A miss of 6 costs 36, nine times as much, though it's only three times as big.
Loss and mean squared error
A loss is one number for how wrong a network is on some examples. Lower is better, and training searches for weights that make it small. Mean squared error, or MSE, is the loss you just worked out: square each error, then average.
MSE = (e₁² + e₂² + … + eₙ²) ÷ n
If you've done "Linear regression" in Classical ML, this is the loss it fits. Its taste for big misses suits jobs where one wild forecast does more harm than many small ones. But one bad reading, such as a sensor that logged 300 °C, can drag the whole fit towards it.
Your turn
Over four days, weather app A misses by 2 degrees every day. App B is spot on for three days and misses by 8 on the fourth. Which has the lower mean squared error?
Part 2. When the answer is a probability
Score a probability with cross-entropy, when the answer is yes and when it's no.
Many networks answer a yes-or-no question with a probability. The XOR network ended in a sigmoid, so its output was a number between 0 and 1: how sure it was that the answer is yes. You could score that with squared error, but a different loss fits probabilities better.
Take the probability the network gave the right answer, call it p, and score −ln(p). ln is the natural log: the ln key on a calculator, math.log in Python. ln(1) = 0, and as p shrinks towards 0, ln(p) falls without limit. So −ln(p) is 0 for a sure right answer and has no ceiling.
Example 2.1
The right answer is yes
A network gives the chance of yes, and the right answer is yes. Score the answers 0.9, 0.6, 0.1 and 0.01 with −ln(p), and with squared error, (1 − p)², beside it.
- Fairly sure and right−ln(0.9) = 0.105; (1 − 0.9)² = 0.01
Both losses are small.
- Right, but only just−ln(0.6) = 0.511; (1 − 0.6)² = 0.16
Both grow.
- 90% sure of the wrong answer−ln(0.1) = 2.303; (1 − 0.1)² = 0.81
A chance of 0.1 for yes means 0.9 for no.
- 99% sure and wrong−ln(0.01) = 4.605; (1 − 0.01)² = 0.98
Squared error barely moves and can never pass 1. The log loss doubles.
Answer−ln(p) climbs from 0.105 to 4.605, while squared error stays below 1.
That's what you want from a classifier. Being 60% sure and wrong is forgivable. Being 99% sure and wrong means something in the weights is badly off, and the loss should say so loudly. A network scored with −ln(p) keeps paying for a confident mistake until it's fixed.
Cross-entropy
The loss −ln(p), where p is the probability the network gave the right answer. Also called log loss. When the right answer is no, p is the probability of no: 1 minus the probability of yes.
loss = −ln(p of the right answer)
One edge: if p is exactly 0, −ln(p) is infinite. PyTorch's BCELoss caps each log at −100 for that reason, and BCEWithLogitsLoss works from the total before the sigmoid, which keeps very confident answers accurate.
Your turn
A network says the chance of yes is 0.8. The right answer is no. What's its cross-entropy?
Part 3. The loss as a surface
Plot the loss against a weight, and read where it's lowest and steepest.
The loss depends on the weights: change a weight and the answers change, so the loss does too. Start with one weight. A shop prices apples by weight, and a one-weight network guesses the price as w × kilos. The data: 1 kg costs 2, 2 kg cost 4, 3 kg cost 6.
Example 3.1
The loss for four values of w
Score the network's guesses with mean squared error for w = 0, 1, 2 and 3. Which w is best?
- Try w = 0guesses 0, 0, 0: (4 + 16 + 36) ÷ 3 = 18.67
Every error is the full price.
- Try w = 1guesses 1, 2, 3: (1 + 4 + 9) ÷ 3 = 4.67
Halfway to the right price, and a quarter of the loss.
- Try w = 2guesses 2, 4, 6: (0 + 0 + 0) ÷ 3 = 0
Every guess is exact.
- Try w = 3guesses 3, 6, 9: (1 + 4 + 9) ÷ 3 = 4.67
Past the best value, the loss climbs again.
Answerw = 2, where the loss is 0.
Every error here is (w − 2) times the kilos, so the loss works out to about 4.67 × (w − 2)²: a bowl. Far from the floor it's steep. Nudge w from 3 to 3.1 and the loss rises by 0.98. Nudge it from 2 to 2.1 and it rises by only 0.05.
Loss surface
The loss for every possible setting of the weights. With one weight it's a curve, with two it's a surface like a bowl, and with millions nobody can draw it. Training looks for a low spot on it.
A network can't see the whole surface, any more than you can see a valley in thick fog. It can only feel the slope where it stands. And with hidden layers the surface stops being a neat bowl: it can have flat stretches and more than one dip.
Your turn
At w = 3, one step past the floor, the loss is 4.67. At w = 4, twice as far from the floor, what is it?
Where you'll meet it
A common question: why cross-entropy, and not squared error, for a classifier? On a probability, squared error never passes 1, so a confident mistake costs little, while −ln(p) has no ceiling. A good sanity check to mention: a fresh classifier that spreads its bets evenly over 10 classes should start at a loss near −ln(0.1) = 2.30.
Language models are trained on cross-entropy, once per token. The model gives a probability to every possible next token, and the loss is −ln of the probability it gave the token that really came next. In PyTorch the losses from this lesson are nn.MSELoss and nn.BCEWithLogitsLoss, and nn.CrossEntropyLoss handles many classes.
import math
def mse(guesses, actual):
return sum((g - a) ** 2 for g, a in zip(guesses, actual)) / len(actual)
print(mse([21, 17, 30], [18, 20, 30])) # 6.0
print(round(-math.log(0.9), 3), round(-math.log(0.01), 3)) # 0.105 4.605
print(round(-math.log(1 - 0.8), 2)) # 1.61
kilos, prices = [1, 2, 3], [2, 4, 6]
losses = [mse([w * k for k in kilos], prices) for w in range(5)]
print([round(x, 2) for x in losses])
# [18.67, 4.67, 0.0, 4.67, 18.67]The last line is the bowl from part 3, at w = 0 to 4. Try w = 1.9 and 2.1 to see how flat it is near the floor.
Check yourself
Three questions from across the lesson. Try each one before you look back.
1
A network answers 100 yes-or-no questions. On 99 it gives the right answer a probability of 0.99. On one it gives the right answer 0.001. Which costs more in total cross-entropy: the 99 good answers together, or the single bad one?
2
One network forecasts tomorrow's electricity use in kilowatt-hours. Another gives the chance that an email is spam. Which losses fit?
3
A network ends in a sigmoid, and its total before the sigmoid is 0. The right answer is yes. What is its cross-entropy?
What to remember
1One number for how wrong
Mean squared error squares each error and averages: forecasts off by 3, −3 and 0 score (9 + 9 + 0) ÷ 3 = 6.
2When the answer is a probability
Cross-entropy is −ln of the probability given to the right answer: 0.105 at 0.9, but 4.605 at 0.01, with no ceiling.
3The loss as a surface
The loss surface is the loss for every setting of the weights; for the apple prices it's a bowl with its floor at w = 2.
Recap and going further
The lesson on one page
- One number, lower is better
- Training searches for weights that shrink it
- Square each error, then average
- One big miss costs more than many small ones
- −ln of the probability given to the right answer
- 0 for a sure right answer, no ceiling for a sure wrong one
- One bad answer can outweigh 99 good ones
- A fresh 10-way classifier should start near 2.30
- The loss for every setting of the weights
- Squared error on one weight makes a bowl
Try this
Compare the two losses in PyTorch
Score the three temperature forecasts with nn.MSELoss. Then score a yes-or-no answer two ways: pass a raw total (a logit) to nn.BCEWithLogitsLoss, and pass the sigmoid of the same total to nn.BCELoss. Try totals of 2, −2, −20 and −200, with yes as the right answer. Last, plot the apple-price network's mean squared error for w from 0 to 4.
You need: Python with PyTorch and matplotlib (pip install torch matplotlib), or a free notebook such as Google Colab.
Look for
- A mean squared error of 6.0 for the three forecasts.
- The total at which the two yes-or-no losses stop agreeing, and why (the
BCELosspage explains it). - Whether the loss levels off or keeps climbing as the total goes from −2 to −200.
- A bowl with its floor at w = 2, getting steeper the further you go.
The words, in plain language
- Loss and mean squared error
- A loss is one number for how wrong a network is on some examples. Lower is better, and training searches for weights that make it small. Mean squared error, or MSE, is the loss you just worked out: square each error, then average.
- Cross-entropy
- The loss −ln(p), where p is the probability the network gave the right answer. Also called log loss. When the right answer is no, p is the probability of no: 1 minus the probability of yes.
- Loss surface
- The loss for every possible setting of the weights. With one weight it's a curve, with two it's a surface like a bowl, and with millions nobody can draw it. Training looks for a low spot on it.
Read next
- Improving the way neural networks learnMichael Nielsen, Neural Networks and Deep Learning, chapter 3Why cross-entropy trains faster than squared error when a sigmoid neuron starts out badly wrong.
- A Recipe for Training Neural NetworksAndrej KarpathyA checklist for training runs. One early check: the loss should start where chance says it will.
- BCEWithLogitsLossPyTorch documentationThe yes-or-no loss that takes raw totals and applies the sigmoid itself, which stays accurate for very confident outputs.
Neural Networks overviewSpotted a mistake? Tell us.