Run a full MLP backward pass on paper

This page runs the training of that machine end to end: the promise, a start that keeps it, the step, the relay that computes the gradient, and the proof on paper. Each node hands the next one exactly what it needs.

flowchart LR
    UAT[UAT promise] --> INIT[break symmetry]
    INIT --> GD[pick direction]
    GD --> BP[backprop relay]
    BP --> SCAL[scalar chain]
    SCAL --> XOR[XOR payoff]

The argument in one line. Promise first, then a start that keeps the promise, then the move, then the machine that computes the move, then the proof on paper.

1 · The words this page uses

Four groups of words carry the rest of the page. Read them once. Each returns later in a worked case.

1.1 · Parameter, weight, bias

A parameter is a number the training step changes. A dense layer holds two kinds. A weight multiplies an input; a bias is added after the sum. For one layer, \(z = Wx + b\): the matrix \(W\) holds the weights and the vector \(b\) the biases. The input \(x\) and the target \(y\) are data. Training never changes them.

1.2 · Activation, loss

An activation is the value a neuron passes on after its nonlinear function. The weighted sum is \(z\), the activation is \(a = g(z)\), and \(g\) is the activation function. The sigmoid used below is \(\sigma(z) = 1/(1+e^{-z})\): it turns any number into a value between 0 and 1. The loss is one number that scores the prediction against the target. Lower is better, and the training step changes the parameters to lower it.

1.3 · Gradient, learning rate

The gradient \(\partial L/\partial w\) is the slope of the loss with respect to one parameter at the current point: how much \(L\) moves per unit change in \(w\). The learning rate \(\eta\) is the step size you choose. The update is \(w \gets w - \eta\,\partial L/\partial w\): the gradient gives the direction, \(\eta\) gives the distance.

1.4 · Epoch, symmetry breaking

An epoch is one pass over the whole training set; you choose how many epochs to run. Symmetry breaking is giving neurons different starting weights so they can learn different features. Identical starts keep the layer from learning distinct features.

2 · Universal Approximation Theorem: the promise

Before training, ask what the machine can compute. One hidden layer with enough neurons approximates any continuous function on a bounded patch of inputs, to any accuracy you demand. That is the whole theorem. Each word carries weight.

Why one layer is this strong. A hidden neuron draws a line. With sigmoid it fires near \(1\) on one side, near \(0\) on the other. The output neuron adds those votes with weights. Strips of fired votes fence off small plateaus, and any continuous shape is a stack of small plateaus. Picture a circle made of square pixels. More neurons give smaller pixels and a sharper shape.

The fine print. The theorem needs a nonlinear activation. Stacked linear layers collapse into one linear map. Nonlinearity at every layer keeps the promise alive.

3 · Weight initialization: break the symmetry

The UAT says good weights exist. Training starts from weights you choose. The obvious start is zero. Zero quietly breaks the promise.

The rule: if a layer's neurons start identical and the next layer treats them alike, they stay identical forever. Identical neurons get the same inputs, compute the same output, receive the same upstream gradient, and take identical updates. A 64-neuron layer with zero init has the capacity of one neuron. Training longer changes nothing.

One caution. Sigmoid does not rescue this. The output layer still learns, and any hidden row that moves moves with the others, so the hidden rows never separate. With ReLU, \(\max(0,z)\), and zero weights the hidden gradient is exactly zero, so the layer does not move at all. Both are the same failure.

The fix is random weights: each neuron starts with a different line, which is exactly what the UAT needs. The scale matters too. Too large and the signal explodes, sigmoid saturates, \(\sigma'(z) \approx 0\), and the gradient dies. Too small and the signal shrinks layer by layer. Two standard recipes hold the signal near one: Xavier draws each weight with variance \(\frac{1}{n_{in}}\), He uses \(\frac{2}{n_{in}}\) for ReLU. The input count appears because a neuron sums \(n_{in}\) contributions — more contributions need smaller weights. PyTorch ships both: nn.init.xavier_normal_ and nn.init.kaiming_normal_.

3.1 · What happens when two units start identical

Predict first. A 2-2-1 network has two hidden rows. Both rows start at zero and both output weights are 1. After one gradient step with \(\eta = 0.1\), will the two rows still match? Then start the rows at random values. Will they match after one step?

import torch
import torch.nn.functional as F

torch.manual_seed(0)
x = torch.tensor([1.0, 2.0])
y = torch.tensor([1.0])

def rows(W):
    return [[round(v, 4) for v in row] for row in W.tolist()]

def one_step(W1_start):
    W1 = W1_start.clone().requires_grad_()
    b1 = torch.zeros(2, requires_grad=True)
    W2 = torch.ones(1, 2, requires_grad=True)   # equal outgoing weights
    b2 = torch.zeros(1, requires_grad=True)
    h = torch.sigmoid(W1 @ x + b1)
    loss = F.binary_cross_entropy_with_logits(W2 @ h + b2, y)
    loss.backward()
    with torch.no_grad():
        for p in (W1, b1, W2, b2):
            p -= 0.1 * p.grad
    return W1.detach()

for name, start in [("zero", torch.zeros(2, 2)), ("random", torch.randn(2, 2))]:
    print(name, "before", rows(start))
    print(name, "after ", rows(one_step(start)))

zero before [[0.0, 0.0], [0.0, 0.0]]
zero after  [[0.0067, 0.0134], [0.0067, 0.0134]]
random before [[1.541, -0.2934], [-2.1788, 0.5684]]
random after  [[1.5465, -0.2825], [-2.1735, 0.5789]]

Equal output weights make the upstream gradient equal for both rows. The zero rows stay equal; the random rows separate.

What happens. The zero rows stay equal. Each gets the same gradient \((-0.0672, -0.1345)\), so both move to \((0.0067, 0.0134)\). The random rows separate, because different rows produce different hidden values, different sigmoid slopes, and different gradients. The output weights must also be equal for the tie to hold; the hands-on changes them and the rows split.

4 · Gradient descent: read the step

You own the update \(w \gets w - \eta\,\partial L/\partial w\). What remains is the trap and a lab skill: read a loss curve. The gradient gives the direction at the current point only. It says nothing about the distance downhill.

One caution. Mini-batch training adds noise — each batch sees different data, so the curve wobbles even when healthy. Read the trend, not single points.

5 · Backpropagation: the graph in reverse

Gradient descent needs \(\partial L/\partial w\) for every weight, and a real network has thousands. Recomputing the loss once per weight is hopeless. Backprop gets every gradient in one sweep.

The forward pass is a graph. For a one-hidden-layer net: \(x \to z_1 = W_1 x + b_1 \to a_1 = \sigma(z_1) \to z_2 = W_2 a_1 + b_2 \to \hat{y} = \sigma(z_2) \to L(\hat{y}, y)\). Each arrow is one computation, one node. The loss sits at the end.

flowchart LR
    X[x] --> Z1[z1] --> A1[a1] --> Z2[z2] --> YH[y-hat] --> L[L]
    L -. backward .-> X

Forward builds every value and keeps it; backward walks the same edges in reverse, rescaling one relay at a time.

The backward rule. Start at the loss and move toward the input. At each node: add all gradients arriving at its output. For each input of the node: multiply that sum by the node's derivative with respect to this input, and send it down. That is your chain rule along a path plus summation at fan-out, applied systematically from the loss backward.

Why the forward values are kept. Each local derivative is written in terms of the forward value. For sigmoid, \(\sigma'(z) = \sigma(z)\,(1-\sigma(z))\). To use it backward, \(z\) must still exist — so forward computes and keeps every intermediate value.

One training step is two passes plus the update: forward to compute and keep every value, backward to apply the rule from the loss, then \(w \gets w - \eta\,\partial L/\partial w\) for every weight at once. One backward sweep costs about as much as one forward pass, which is why training is practical at all.

5.1 · Flow versus storage

Flow covers every node, activations included — the gradient reaches the weights only through the activations. Storage covers only the parameters: PyTorch saves .grad on leaves (weights and biases) because the update reads only those. Activation gradients are computed, used, and dropped. Both facts hold at once.

import torch

w = torch.tensor([2.0], requires_grad=True)   # a leaf parameter
x = torch.tensor([1.0])
z = w * x                                     # intermediate value
a = torch.sigmoid(z)                          # intermediate value
loss = (a - 0.5) ** 2
loss.backward()

print(w.grad)   # tensor([0.0799]) — stored, w is a leaf
print(a.grad)   # None — computed on the way, then dropped
print(z.grad)   # None — computed on the way, then dropped

The backward pass used the gradients at z and a (roughly 0.76 × 0.105 ≈ 0.08) to reach w, then kept only the leaf.

6 · Backward pass by hand: a scalar network

The rule above is abstract until you run it with numbers. This section is the recipe plus its result. After it, loss.backward() is a machine doing what you just did by hand.

The recipe. Draw the graph left to right, one node per arithmetic step. Forward: fill every value, keep them all. Turn around at the loss. At each node ask what gradient arrived from above and how much your output wiggles when your input wiggles. Multiply the two and pass the result down. At fan-out, sum the arriving gradients first. Stop at every weight and bias.

The network. One input, one hidden neuron, one output neuron, no fan-out yet. \(x = 1.0\), \(z_1 = w_1 x + b_1\), \(a_1 = \sigma(z_1)\), \(z_2 = w_2 a_1 + b_2\), \(\hat{y} = \sigma(z_2)\). Parameters \(w_1 = 0.5\), \(b_1 = 0\), \(w_2 = 1.0\), \(b_2 = 0\), target \(y = 0\). Loss is Huber with \(\delta = 0.5\): quadratic within the band, flat slope beyond it.

Forward pass — compute and keep every value. The error \(0.6508\) exceeds \(\delta = 0.5\), so the Huber slope is \(\delta \cdot \mathrm{sign}(e) = 0.5\).

Backward pass — start at the loss, multiply by each local derivative on the way down.

Watch two rows get built. Row \(z_2\): arriving \(0.5000\), local \(0.6508 \times 0.3492 = 0.2273\), product \(0.1136\). Row \(w_2\): arriving \(0.1136\), local \(a_1 = 0.6225\), product \(0.0707\). Every other row is the same two questions and one multiply. Bias rows ride the same chain with one factor less, because \(+b\) has derivative \(1\).

6.1 · Check the hand pass with torch

Predict first. The same graph and the same numbers, built in torch. Before you run it, write the six values the code should print: \(\hat{y}\), \(L\), and the gradients at \(w_2\), \(b_2\), \(w_1\), \(b_1\).

import torch
import torch.nn.functional as F

x  = torch.tensor(1.0)
w1 = torch.tensor(0.5, requires_grad=True)
b1 = torch.tensor(0.0, requires_grad=True)
w2 = torch.tensor(1.0, requires_grad=True)
b2 = torch.tensor(0.0, requires_grad=True)

z1   = w1 * x + b1
a1   = torch.sigmoid(z1)
z2   = w2 * a1 + b2
yhat = torch.sigmoid(z2)
loss = F.huber_loss(yhat, torch.tensor(0.0), delta=0.5)
loss.backward()

print("yhat", round(yhat.item(), 4))         # yhat 0.6508
print("loss", round(loss.item(), 4))         # loss 0.2004
print("w2.grad", round(w2.grad.item(), 4))   # w2.grad 0.0707
print("b2.grad", round(b2.grad.item(), 4))   # b2.grad 0.1136
print("w1.grad", round(w1.grad.item(), 4))   # w1.grad 0.0267
print("b1.grad", round(b1.grad.item(), 4))   # b1.grad 0.0267

The code follows the table's graph. backward() walks it in reverse; each .grad is the product of the local derivatives on the path from the loss.

What happens. All six values match the tables to four decimals. \(b_1\) and \(w_1\) share a gradient here because \(x = 1\), so \(\partial z_1/\partial b_1 = \partial z_1/\partial w_1 = 1\). PyTorch is not using a different rule; it is running the same chain rule on the same graph.

7 · XOR: the payoff

XOR outputs 1 when the inputs differ: (0,0) → 0, (0,1) → 1, (1,0) → 1, (1,1) → 0. A linear model computes \(z = w_1 x_1 + w_2 x_2 + b\) and thresholds it — one straight line. No line separates this table: (0,0) forces \(b < 0\), the two singletons force \(w_1, w_2 > -b\), and then (1,1) lands above threshold while the table says 0. Contradiction. The two diagonals of the square cannot be split by any single line. XOR is not linearly separable.

A hidden layer fixes it with two lines instead of one. One hidden neuron learns OR, another learns AND, and the output computes OR minus AND — exactly XOR. The UAT promise in miniature: more lines, more shapes.

The machine. Two inputs, two hidden sigmoid neurons, one output sigmoid neuron, binary cross-entropy (BCE) loss. For one example, \(L = -[\, y \ln \hat{y} + (1-y)\ln(1-\hat{y}) \,]\) with \(\hat{y} = \sigma(z_3)\). Differentiate and multiply by the sigmoid derivative and the denominator cancels exactly: \(\frac{dL}{dz_3} = \hat{y} - y\), prediction minus target. Remember this; it saves you in debugging.

Nine parameters: \(W_1\) has \(2 \times 2 = 4\), \(b_1\) has 2, \(W_2\) has \(1 \times 2 = 2\), and \(b_2\) has 1. The count is small enough to check by hand.

One numeric pass. Input \(x = (1, 0)\), target \(y = 1\). Hidden 1: \(w = (1.0, 0.5)\), \(b = 0\). Hidden 2: \(w = (-0.5, 1.0)\), \(b = 0.5\). Output: \(w = (1.0, 1.0)\), \(b = 0\).

Hidden weights multiply by their inputs: \(\partial L/\partial w_{1,1} = -0.0444 \times 1 = -0.0444\), \(\partial L/\partial w_{1,2} = 0\), \(\partial L/\partial w_{2,1} = -0.0565\), \(\partial L/\partial w_{2,2} = 0\). The bias gradients are the pre-activation gradients, \(b_1\): \(-0.0444\) and \(b_2\): \(-0.0565\). Fan-out enters at the input \(x_1\): it feeds two hidden neurons, so its gradient sums both paths, \(-0.0444 \times 1.0 + (-0.0565) \times (-0.5) = -0.0162\). Weight gradients never sum — each weight sits on exactly one path.

Hands-on

1. Sketch the bump \(g(x) = \sigma(10(x-1)) - \sigma(10(x-2))\). Predict its height and width before you plot it, then check: it is about one unit tall and one unit wide. This is the sigmoid step the UAT stacks.
2. Run the §3.1 experiment. Then change W2 to torch.tensor([[1.0, -1.0]], requires_grad=True) and run again. Predict whether the two hidden rows still match. They split, because the upstream gradients now differ; identical rows alone do not keep the tie when the next layer treats them differently.
3. Run \(w \gets w - 1.1 \times 2w\) from \(w=3\) for three steps. Predict whether the loss falls after the second step. It does not: 9, 12.96, 18.66, 26.87. The gradient changes sign with \(w\), and the fixed step keeps overshooting.
4. Run the §5.1 snippet. Predict which of w.grad, a.grad, z.grad is None before you run it. Check 0.0799 on the leaf; the other two are None.
5. Rebuild §6's graph in torch. Predict each of the six printed values, then run the code in §6.1. Change \(x\) to 2 and predict whether \(\partial L/\partial w_1\) doubles before you rerun; it does not, because \(z_1 = 1\) changes the sigmoid slope.
6. Count the 2-2-1 parameters from §7: \(4 + 2 + 2 + 1 = 9\). Then run one SGD step with \(\eta = 0.1\) on the XOR weights. Predict the sign of each weight change from the gradient table before you run it.

One-line understanding. A wide-enough net can draw the shape, a broken-symmetry start lets it, and backprop walks the graph backward multiplying local rates and summing where paths split.