Neural Networks, lesson 1
One neuron
Inputs times weights, plus a bias, through an activation function.
Every network in this track, from a four-neuron toy to a chatbot, is built from one small part used over and over. It takes in a few numbers and sends out one, and this lesson takes it apart. You need to multiply and add, and to read a little Python.
By the end, you can:
Part 1. Each input gets a weight
Work out a neuron's total from its inputs, weights and bias.
A neuron reads a few numbers and keeps a few of its own: one for each input, saying how much that input counts, and one extra number added at the end. To answer, it multiplies each input by its own number and adds everything up. Here's one with two inputs.
Example 1.1
Two inputs, one total
A neuron reads x₁ = 2.0 and x₂ = −1.0. It keeps w₁ = 0.5 for the first input, w₂ = 1.0 for the second, and b = 0.2 to add at the end. What total does it work out?
- Weigh the first input2.0 × 0.5 = 1.0
w₁ goes with x₁. A weight under 1 shrinks its input.
- Weigh the second input−1.0 × 1.0 = −1.0
A negative product pulls the total down.
- Add the products, then b1.0 + (−1.0) + 0.2 = 0.2
The two inputs cancel out exactly, so only b is left.
AnswerA total of 0.2.
Each product is one input's say in the answer. A bigger weight makes that input count for more. A negative weight turns the input into a push the other way. A weight of 0 takes away its say completely, whatever value the input has.
The extra number is what the neuron answers when every input is 0. It lets the neuron lean one way before it has read anything.
Weights and bias
A weight sets how much one input counts, and in which direction. The bias is added at the end, so the neuron can lean one way even when every input is 0. Training is the search for good values of both.
total = w₁ × x₁ + w₂ × x₂ + b
If you've done "Linear regression" in Classical ML, you've met this sum already: a linear regression is one neuron with nothing after the total. The usual slips are leaving out the bias and losing a minus sign. A negative input times a positive weight pulls the total down.
Your turn
A neuron reads three inputs, 1.0, 3.0 and −2.0, with weights 0.4, −0.5 and 0.25, and a bias of 1.0. What is its total?
Part 2. The activation function
Pass a total through ReLU, and say what comes out.
If a neuron stopped at its total, it could only ever add things up. One more step changes what it can do: the total goes through a small, fixed rule before it leaves. The most common rule is almost too simple. If the total is below zero, send out 0. Otherwise, send the total out unchanged.
Example 2.1
The same neuron, two inputs
The neuron from part 1 (weights 0.5 and 1.0, bias 0.2) now sends its total through that rule. What does it send out for the inputs (2.0, −1.0), and for (−2.0, 0.5)?
- Total for the first input2.0 × 0.5 + (−1.0) × 1.0 + 0.2 = 0.2
The total from part 1.
- Apply the rule0.2 → 0.2
It's above zero, so it passes unchanged.
- Total for the second input(−2.0) × 0.5 + 0.5 × 1.0 + 0.2 = −0.3
−1.0 + 0.5 + 0.2.
- Apply the rule−0.3 → 0
Any total below zero comes out as 0, however far below it is.
Answer0.2 for the first input, 0 for the second.
The rule turns the neuron into a switch with a dial. On one side of zero it's off and says nothing. On the other side it passes on how far past zero the total went. From outside, a total of −0.3 and a total of −70 look the same: both send out 0.
Activation function and ReLU
An activation function is the fixed rule a neuron applies to its total before passing it on. ReLU, short for rectified linear unit, is the one above: keep a positive total, and send out 0 for anything else.
ReLU(total) = max(0, total)
ReLU isn't the only choice. Sigmoid squeezes any total into the range 0 to 1, and tanh squeezes it into −1 to 1. Both are smooth curves with no corner, and the next lesson uses sigmoid. And ReLU is not "drop the minus sign": a total of −2 sends out 0, not 2.
Your turn
A ReLU neuron has weights 2.0 and −1.0 and a bias of −1.0. What does it send out for the inputs (1.0, 3.0), and for (2.0, 1.0)?
Part 3. A job no straight line can do
Build a V-shaped answer from two ReLU neurons.
A thermostat wants to know how far the room is from 20 °C, either way: 5 at 25 °C, 6 at 14 °C, 0 at 20 °C. A neuron with no activation works out w × t + b, a straight line that rises or falls at one fixed rate. This job falls until 20 °C, then rises.
Example 3.1
Distance from 20 °C
The too-hot neuron has weight 1 and bias −20. The too-cold neuron has weight −1 and bias 20. Both use ReLU, and the output adds what they send out. What's the output at 25 °C?
- Run the too-hot neuron1 × 25 − 20 = 5 → 5
Above zero, so it passes.
- Run the too-cold neuron−1 × 25 + 20 = −5 → 0
Below zero, so ReLU sends out 0.
- Add the two5 + 0 = 5
At 14 °C they swap roles: 0 + 6 = 6.
Answer5, the room's distance from 20 °C.
Each neuron stays quiet on one side of 20 and counts on the other. Take ReLU out and at 25 °C you get 5 + (−5) = 0. Every temperature gives 0: adding straight lines only ever gives another straight line, here a flat one.
More layers don't help. A layer with no activation only multiplies and adds, so two such layers fold into one: 2 × (3t + 1) − 5 is just 6t − 3. Ten of them still make one straight line.
Linear
A function is linear when its graph is a straight line, like w × t + b. Add linear functions, or feed one into another, and you get another linear one. An activation between layers stops the folding.
2 × (3t + 1) − 5 = 6t − 3
Many ReLU neurons side by side, each switching on at a different point, can add up to far more than a V. In 1989 George Cybenko proved that one layer of enough sigmoid neurons can match any continuous curve on a fixed range as closely as you like.
In 1993 Moshe Leshno and colleagues showed the same for ReLU. The catch is the word "enough": it can be a huge number of neurons.
Your turn
A thermostat aims for 22 °C. Its two ReLU neurons work out ReLU(t − 22) and ReLU(22 − t), and the output adds them. What's the output at 17 °C?
Where you'll meet it
Two questions from this lesson come up often in ML interviews. What does a deep network with no activation functions compute? One linear function, however many layers it has. What does a ReLU neuron send out for a negative total? 0, for every negative total. A later lesson shows how that can leave a neuron stuck at 0 for good.
Print a PyTorch model and you'll see lines like Linear(in_features=512, out_features=2048, bias=True) then ReLU(). That's 2,048 neurons, each with 512 weights and a bias, followed by ReLU. The original 2017 transformer, the design behind today's chatbots, has that pair in every layer: max(0, xW₁ + b₁).
def neuron(inputs, weights, bias):
total = sum(x * w for x, w in zip(inputs, weights)) + bias
return max(0.0, total) # ReLU
print(neuron([2.0, -1.0], [0.5, 1.0], 0.2)) # 0.2
print(neuron([-2.0, 0.5], [0.5, 1.0], 0.2)) # 0.0
for t in [14.0, 20.0, 25.0]:
out = neuron([t], [1.0], -20.0) + neuron([t], [-1.0], 20.0)
print(t, out)
# 14.0 6.0
# 20.0 0.0
# 25.0 5.0Change max(0.0, total) to just total and every output of the loop becomes 0.0: without ReLU, the V goes flat.
Check yourself
Three questions from across the lesson. Try each one before you look back.
1
A neuron works out w₁ × x₁ + w₂ × x₂ + b. You want it to ignore x₂ completely, whatever value x₂ takes. Which one change does that?
2
A ReLU neuron has weights 1.0 and −2.0 and a bias of 0.5. What does it send out for the inputs (3.0, 2.0)?
3
You stack ten layers of neurons and leave out the activation function everywhere. What can the whole network compute?
What to remember
1Each input gets a weight
A neuron multiplies each input by its weight, adds the products, then adds the bias: 2.0 × 0.5 + (−1.0) × 1.0 + 0.2 = 0.2.
2The activation function
ReLU sends out the total when it's above zero and 0 when it isn't: a total of 0.2 sends out 0.2, and −0.3 sends out 0.
3A job no straight line can do
Layers with no activation fold into one straight line, as 2 × (3t + 1) − 5 = 6t − 3, while two ReLU neurons can make a V.
Recap and going further
The lesson on one page
- Each input times its own weight
- Bigger counts for more, negative pushes the other way
- A weight of 0 ignores that input
- Added after the products
- The total when every input is 0
- max(0, total)
- Off below zero, passes the total above it
- Layers with no activation fold into one straight line
- Two ReLU neurons make a V that no line can
- Enough of them can match any continuous curve
- Enough can be a very large number
Try this
Watch a network stay straight
Open TensorFlow Playground (playground.tensorflow.org) and pick the circle dataset, where one colour sits inside a ring of the other. Set the activation to Linear, add four hidden layers of six neurons, and press play. Then switch the activation to ReLU, and after that to Tanh. Last, go back to one hidden layer and find the fewest ReLU neurons that can still fence in the middle.
You need: A web browser. TensorFlow Playground runs entirely in the page, with nothing to install.
Look for
- Whether the boundary ever bends with Linear activation, however many layers you add.
- How the ReLU boundary is made of straight pieces with corners, while the Tanh one is smooth.
- The smallest hidden layer of ReLU neurons that encloses the centre.
- Which connecting lines turn blue (positive weights) and which orange (negative) as it trains.
The words, in plain language
- Weights and bias
- A weight sets how much one input counts, and in which direction. The bias is added at the end, so the neuron can lean one way even when every input is 0. Training is the search for good values of both.
- Activation function and ReLU
- An activation function is the fixed rule a neuron applies to its total before passing it on. ReLU, short for rectified linear unit, is the one above: keep a positive total, and send out 0 for anything else.
- Linear
- A function is linear when its graph is a straight line, like w × t + b. Add linear functions, or feed one into another, and you get another linear one. An activation between layers stops the folding.
Read next
- Neural Networks Part 1: Setting up the ArchitectureStanford CS231n notesThe neuron model, the common activation functions, and how ReLU compares with sigmoid and tanh.
- A visual proof that neural nets can compute any functionMichael Nielsen, Neural Networks and Deep LearningBuilds any curve out of many simple neurons, one step at a time, with sliders to play with.
Neural Networks overviewSpotted a mistake? Tell us.