Neural Networks, lesson 2
Stacking neurons into layers
Why one neuron can't solve XOR, and how a hidden layer can.
One neuron draws one straight line through its inputs, and some jobs need more than one line. This lesson finds such a job, solves it with a second layer of neurons, and counts the numbers a network stores. You need the weighted sum and ReLU from "One neuron".
By the end, you can:
Part 1. Where one line runs out
Tell whether one neuron can split a yes-or-no job.
Guess first
XOR says yes when exactly one of its two inputs is 1. You train a single neuron with a yes/no cut on XOR for a million rounds, taking tiny steps. What happens?
Give a neuron a yes-or-no job: yes when its total is above zero, no when it isn't. Put x₁ along the bottom and x₂ up the side. Where the total is exactly zero you get a straight line, with yes on one side and no on the other.
Example 1.1
A cut for AND
AND says yes only when both inputs are 1. Does a neuron with weights 1 and 1 and bias −1.5 get all four input pairs right?
- Try (0, 0)0 + 0 − 1.5 = −1.5 → no
AND says no. Right.
- Try (0, 1) and (1, 0)1 − 1.5 = −0.5 → no
AND says no to both. Right.
- Try (1, 1)1 + 1 − 1.5 = 0.5 → yes
Only both together get past the bias. Right.
AnswerYes: all four come out right.
The total is zero along x₁ + x₂ = 1.5, a line that cuts the top-right corner off the square. The weights tilt that line and the bias slides it, but whatever the numbers, one neuron splits yes from no with one straight line.
Linearly separable
A yes-or-no job is linearly separable when one straight line (or, with more inputs, one flat cut) can put every yes on one side and every no on the other. AND and OR are. XOR isn't.
Example 1.2
No cut for XOR
Can any weights w₁ and w₂ and bias b say yes to (0, 1) and (1, 0), and no to (0, 0) and (1, 1)?
- (0, 0) must say nob ≤ 0
Its total is just b.
- The two yes casesw₁ + b > 0 and w₂ + b > 0
(1, 0) and (0, 1) need totals above zero.
- Add those twow₁ + w₂ + 2b > 0, so w₁ + w₂ + b > −b ≥ 0
That's the total for (1, 1), so it says yes. XOR needs a no.
AnswerNo weights and bias can do it.
Your turn
OR says yes when at least one input is 1. Which weights and bias make one neuron do OR?
Part 2. A hidden layer solves XOR
Run a two-layer network by hand, one layer at a time.
The fix is more neurons. XOR is really "OR, but not both". So one neuron can do OR, a second can do NAND ("not both"), and a third can say yes only when both of those say yes, which is AND.
These neurons use sigmoid as their activation function. It squeezes any total into the range 0 to 1: a total of 10 comes out as 0.99995, and −10 as 0.00005. With weights as big as 20, every total lands far from zero, so each answer is almost exactly 0 or 1.
Example 2.1
XOR of 1 and 1
Neuron h₁ (OR) works out 20x₁ + 20x₂ − 10. Neuron h₂ (NAND) works out −20x₁ − 20x₂ + 30. The output (AND) works out 20h₁ + 20h₂ − 30. Each total goes through sigmoid. What does the network say for (1, 1)?
- Run h₁ (OR)20 + 20 − 10 = 30 → 1.00
At least one input is on.
- Run h₂ (NAND)−20 − 20 + 30 = −10 → 0.00
Both inputs are on, so NAND says no.
- Run the output (AND)20 × 1.00 + 20 × 0.00 − 30 = −10 → 0.00
Its inputs are h₁ and h₂, not x₁ and x₂. One yes isn't enough for AND.
Answer0, as XOR of 1 and 1 should be.
Each middle neuron still draws one straight line. The output neuron combines their answers, so the network can say yes inside a band between two lines, a shape no single line makes. Running the layers in order, with each layer's outputs as the next one's inputs, is how every network answers.
Hidden layer and forward pass
A hidden layer sits between the inputs and the output. Its answers never appear in the data: the network makes up its own in-between questions, here "is either on?" and "are both on?". A forward pass is one run from inputs to output, a layer at a time.
XOR = AND(OR, NAND)
These weights were picked by hand. A trained network finds its own, and they rarely look this tidy: it may split the job into quite different in-between questions. How it finds them is what the rest of this module is about.
Your turn
Same network, input (0, 0). What do h₁, h₂ and the output send out?
Part 3. Counting parameters
Count a network's parameters, and give each layer's weights their shape.
Every weight and every bias is a number training can change, and a network stores nothing else. The XOR network has 9 of them: each hidden neuron has 2 weights and a bias, and so does the output. When you read about an 8-billion-parameter model, those are its 8 billion numbers.
Example 3.1
A digit reader
A classic network reads a 28 × 28 pixel image of a handwritten digit, so 784 inputs. It has one hidden layer of 128 neurons and 10 outputs, one per digit. How many numbers does training set?
- Weights into the hidden layer784 × 128 = 100,352
Each of the 128 neurons has one weight per pixel.
- Hidden biases128
One per neuron.
- The output layer128 × 10 + 10 = 1,290
Each output reads the 128 hidden neurons, and has a bias.
- Add them up100,352 + 128 + 1,290 = 101,770
Almost all of them sit in the first layer.
Answer101,770 numbers.
Each neuron has one weight for every input it reads, plus its bias. So a layer of n neurons reading m inputs holds n × (m + 1) of them. The inputs have none of their own: the 784 pixels are data, not something training changes.
Parameters
Every weight and every bias in a network: the numbers training changes. A layer holds its number of neurons times one more than its number of inputs.
parameters in a layer = neurons × (inputs + 1)
Code stores a layer's weights as a grid: one row per neuron, one column per input. The digit reader's hidden layer is a grid of shape (128, 784), plus 128 biases. A shape error usually means two layers don't line up: one sends out a different number of values than the next one reads.
Your turn
A network reads 3 inputs, has a hidden layer of 4 neurons, and has 2 outputs. How many parameters does it have?
Where you'll meet it
"Why can't a single neuron learn XOR?" is a classic early question: one neuron draws one straight line, and XOR's yes cases sit on a diagonal. A common follow-up asks you to count a layer's parameters, or to find the layer that doesn't line up in a broken network. Say neurons × (inputs + 1) out loud as you go.
In PyTorch, nn.Linear(784, 128) is the digit reader's hidden layer. Its weight has shape (128, 784), one row per neuron, and its bias holds 128 numbers. sum(p.numel() for p in model.parameters()) counts every parameter: 101,770 for the whole digit reader. 3Blue1Brown's digit network, with two hidden layers of 16, has 13,002.
import math
def layer(inputs, weights, biases):
# one row of weights per neuron, then sigmoid
return [1 / (1 + math.exp(-(sum(w * x for w, x in zip(row, inputs)) + b)))
for row, b in zip(weights, biases)]
for x in [(0, 0), (0, 1), (1, 0), (1, 1)]:
h = layer(x, [[20, 20], [-20, -20]], [-10, 30])
out = layer(h, [[20, 20]], [-30])
print(x, round(out[0], 3))
# (0, 0) 0.0
# (0, 1) 1.0
# (1, 0) 1.0
# (1, 1) 0.0Each layer call is one layer of the forward pass, and zip(weights, biases) walks the grid one row, so one neuron, at a time.
Check yourself
Three questions from across the lesson. Try each one before you look back.
1
Which of these yes-or-no jobs can a single neuron do?
2
Two ReLU neurons read x = (1, 2). The first has weights (1, −1) and bias 0; the second has weights (1, 1) and bias −1. The output has weights (2, 1), bias 0 and no activation. What does the network send out?
3
Code stores a layer's weights as a grid with one row per neuron and one column per input. A layer of 3 neurons reads from a layer of 2. What shape is its weight grid?
What to remember
1Where one line runs out
One neuron splits its inputs with one straight line, so it can do AND (weights 1 and 1, bias −1.5) but never XOR.
2A hidden layer solves XOR
A hidden layer lets several lines work together: XOR = AND(OR, NAND), worked out one layer at a time in a forward pass.
3Counting parameters
A layer of n neurons reading m inputs has n × (m + 1) parameters, so a 784 → 128 → 10 network has 101,770.
Recap and going further
The lesson on one page
- Its yes and no are split by one straight cut
- AND and OR can be split that way; XOR can't
- Each neuron draws its own line
- XOR is AND of OR and NAND
- Run one layer, feed its outputs to the next
- The weights stay fixed while the numbers flow through
- Every weight and every bias
- Per layer: neurons × (inputs + 1)
- Weight grid: one row per neuron, one column per input
- A shape error means two layers don't line up
Try this
Build the XOR network in PyTorch
Make nn.Sequential(nn.Linear(2, 2), nn.Sigmoid(), nn.Linear(2, 1), nn.Sigmoid()). Inside torch.no_grad(), copy this lesson's hand-picked weights and biases into the two Linear layers, then run all four corners through it. Print each layer's weight.shape and the total from sum(p.numel() for p in model.parameters()). Then build a 784 → 128 → 10 network and count again.
You need: Python with PyTorch (pip install torch), or a free notebook such as Google Colab.
Look for
- Outputs close to 0, 1, 1 and 0 for the four corners.
- Weight shapes of (2, 2) and (1, 2): one row per neuron, one column per input.
- A total of 9 for the XOR network and 101,770 for the digit reader.
- The error PyTorch gives if the first layer sends out 3 numbers and the second still reads 2.
The words, in plain language
- Linearly separable
- A yes-or-no job is linearly separable when one straight line (or, with more inputs, one flat cut) can put every yes on one side and every no on the other. AND and OR are. XOR isn't.
- Hidden layer and forward pass
- A hidden layer sits between the inputs and the output. Its answers never appear in the data: the network makes up its own in-between questions, here "is either on?" and "are both on?". A forward pass is one run from inputs to output, a layer at a time.
- Parameters
- Every weight and every bias in a network: the numbers training changes. A layer holds its number of neurons times one more than its number of inputs.
Neural Networks overviewSpotted a mistake? Tell us.