The Chain Rule — Heart of Backpropagation

The chain rule is the single most important calculus concept for deep learning. Backpropagation is nothing but the chain rule applied to a computation graph.

For a composition F(x) = f(g(x)):
F'(x) = f'(g(x)) · g'(x)

In Leibniz notation (dg "cancels"):
dF/dx = (df/dg) · (dg/dx)

For long compositions (deep networks):
d/dx[f(g(h(u(v(x)))))] = (df/dg) · (dg/dh) · (dh/du) · (du/dv) · (dv/dx)

Each layer contributes one term to the product.

Multivariable chain rule (from Raschka's appendix):
d/dx[f(g(x), h(x))] = (∂f/∂g)(dg/dx) + (∂f/∂h)(dh/dx)

In vector form: ∇f · v'(x)

Vanishing gradients: when many terms in this product are small (< 1), their product → 0. Gradients in early layers become tiny — the network stops learning. This is why:
• Deep networks with sigmoid activations don't train
• ReLU doesn't saturate → no vanishing gradient
• Residual connections add a "gradient highway"

1import numpy as np
2from sympy import *
3
4x = symbols('x')
5
6# ── Chain rule: Raschka's example f(x) = log(sqrt(x)) ────────────
7print("Chain rule: f(x) = log(sqrt(x))")
8print("Decompose: g(x) = sqrt(x), f(g) = log(g)")
9print()
10
11# Step by step
12g     = sqrt(x)
13f_g   = log(g)
14g_sym = symbols('g')
15
16d_outer = diff(log(g_sym), g_sym)   # df/dg = 1/g
17d_inner = diff(sqrt(x), x)          # dg/dx = 1/(2√x)
18result  = simplify(diff(log(sqrt(x)), x))
19
20print(f"  df/dg (outer) = {d_outer} = 1/√x when substituting")
21print(f"  dg/dx (inner) = {d_inner}")
22print(f"  df/dx = (df/dg)(dg/dx) = {result}")
23
24# ── Backprop through a simple 2-layer network ─────────────────────
25print("
26Backprop: 2-layer network (forward and backward pass)")
27np.random.seed(42)
28
29# One sample, one output
30x_data = np.array([[1.0, 2.0, 3.0]])   # (1, 3)
31y_data = np.array([[1.0]])              # (1, 1)
32
33W1 = np.random.randn(3, 4) * 0.1   # (3, 4)
34b1 = np.zeros((1, 4))
35W2 = np.random.randn(4, 1) * 0.1   # (4, 1)
36b2 = np.zeros((1, 1))
37
38def relu(z): return np.maximum(0, z)
39def relu_grad(z): return (z > 0).astype(float)
40
41# FORWARD PASS — store intermediates for backprop
42z1 = x_data @ W1 + b1     # (1, 4)
43a1 = relu(z1)              # (1, 4)
44z2 = a1 @ W2 + b2         # (1, 1)
45loss = 0.5 * np.sum((z2 - y_data)**2)
46
47print(f"  Loss: {loss:.4f}")
48
49# BACKWARD PASS — chain rule at each layer
50dL_dz2 = z2 - y_data                    # dL/dz2 = (ŷ-y)
51dL_dW2 = a1.T @ dL_dz2                  # dL/dW2 = a1ᵀ · dL/dz2
52dL_db2 = dL_dz2
53
54dL_da1 = dL_dz2 @ W2.T                  # dL/da1 = dL/dz2 · W2ᵀ
55dL_dz1 = dL_da1 * relu_grad(z1)         # chain rule through ReLU
56dL_dW1 = x_data.T @ dL_dz1             # dL/dW1
57dL_db1 = dL_dz1
58
59print(f"  ||dL/dW2|| = {np.linalg.norm(dL_dW2):.4f}")
60print(f"  ||dL/dW1|| = {np.linalg.norm(dL_dW1):.4f}")
61
62# ── Vanishing gradient demo ───────────────────────────────────────
63def sigmoid(x): return 1 / (1 + np.exp(-x))
64def sigmoid_grad(x): return sigmoid(x) * (1 - sigmoid(x))
65
66# Maximum gradient of sigmoid is 0.25 (at x=0)
67# Through 10 layers: 0.25^10 ≈ 0.000001 ← vanishes!
68x_val = 0.0
69grad  = sigmoid_grad(x_val)
70print(f"
71Vanishing gradient:")
72print(f"  Max sigmoid gradient: {grad:.4f}")
73for layers in [1, 3, 5, 10]:
74    vanished = grad ** layers
75    print(f"  Through {layers:2d} layers: {vanished:.8f}")