DEEP LEARNING / 2. BACKPROPAGATION

Backpropagation

How networks actually learn — chain rule applied recursively

EXPLANATION

Backpropagation is just the chain rule of calculus applied to a computation graph. Nothing more.

The goal: compute ∂Loss/∂w for every weight w in the network. Then update: w = w - lr * ∂Loss/∂w

Steps:
1. Forward pass  → compute predictions, store intermediate values
2. Compute loss  → how wrong are we? (MSE, CrossEntropy, etc.)
3. Backward pass → compute gradients layer by layer, from output to input
4. Update        → gradient descent step

PyTorch does all of this automatically via autograd. You just call loss.backward() and it computes all gradients. But understanding what's happening under the hood is essential for debugging and designing architectures.

DATA FLOW

FORWARD PASS (left to right, store values):
  x → [Layer1] → h1 → [Layer2] → h2 → [Layer3] → ŷ → Loss

  BACKWARD PASS (right to left, chain rule):
  ∂L/∂w3 = ∂L/∂ŷ · ∂ŷ/∂w3
  ∂L/∂w2 = ∂L/∂ŷ · ∂ŷ/∂h2 · ∂h2/∂w2
  ∂L/∂w1 = ∂L/∂ŷ · ∂ŷ/∂h2 · ∂h2/∂h1 · ∂h1/∂w1

  Each gradient = product of all gradients downstream
  PyTorch: loss.backward() does this automatically

CODE

PYTHON
1import torch
2import torch.nn as nn
3
4# ── Autograd: how PyTorch tracks gradients ────────────────────────
5x = torch.tensor(2.0, requires_grad=True)
6y = x ** 3 + 2 * x           # y = x³ + 2x
7y.backward()                  # dy/dx = 3x² + 2
8print(f"dy/dx at x=2: {x.grad}")  # = 3(4) + 2 = 14
9
10# ── Full training loop — manual to understand what's happening ────
11torch.manual_seed(42)
12
13# Toy dataset: y = 2x + 1
14X = torch.randn(100, 1)
15y = 2 * X + 1 + 0.1 * torch.randn(100, 1)   # with noise
16
17# Model: one linear layer
18model = nn.Linear(1, 1)
19optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
20loss_fn = nn.MSELoss()
21
22print(f"Before: w={model.weight.item():.4f}, b={model.bias.item():.4f}")
23
24for epoch in range(100):
25    # 1. Forward pass
26    y_pred = model(X)
27
28    # 2. Loss
29    loss = loss_fn(y_pred, y)
30
31    # 3. Zero gradients (important! gradients accumulate by default)
32    optimizer.zero_grad()
33
34    # 4. Backward pass — computes ∂loss/∂w and ∂loss/∂b
35    loss.backward()
36
37    # 5. Update weights: w = w - lr * grad
38    optimizer.step()
39
40    if epoch % 20 == 0:
41        print(f"Epoch {epoch:3d} | Loss: {loss.item():.6f}")
42
43print(f"After:  w={model.weight.item():.4f}, b={model.bias.item():.4f}")
44# Should be close to w=2.0, b=1.0
← PREV1. Perceptron & ActivationsNEXT →3. Optimizers