CALCULUS / 3. DIFFERENTIATION RULES

Differentiation Rules

Sum, product, quotient, chain — the toolkit for any function

EXPLANATION

You rarely compute derivatives from the limit definition. Instead you use rules that combine simpler derivatives.

From Table 2 of Raschka's calculus appendix:

Sum rule:       d/dx[f+g] = f' + g'
Difference rule: d/dx[f-g] = f' - g'
Product rule:   d/dx[f·g] = f'·g + f·g'
Quotient rule:  d/dx[f/g] = [g·f' - f·g'] / g²
Chain rule:     d/dx[f(g(x))] = f'(g(x)) · g'(x)

Power rule:     d/dx[xⁿ] = n·xⁿ⁻¹
Constant rule:  d/dx[c] = 0

The chain rule is the most important for deep learning — it's what makes backpropagation work. Every layer in a neural network is a composition of functions, and the chain rule tells you how to differentiate through them.

In Leibniz notation, the chain rule looks like fractions canceling:
df/dx = (df/dg) · (dg/dx)

The dg "cancels" — this makes it easy to remember and apply to long chains of composed functions.

DIAGRAM

Chain rule example: f(x) = log(√x)
  Decompose: g(x) = √x,  f(g) = log(g)

  Step 1: derivative of outer w.r.t. inner
    df/dg = d/dg[log(g)] = 1/g = 1/√x

  Step 2: derivative of inner w.r.t. x
    dg/dx = d/dx[x^(1/2)] = (1/2)x^(-1/2) = 1/(2√x)

  Step 3: multiply
    df/dx = (1/√x) · (1/(2√x)) = 1/(2x)

  Neural network chain:
  Loss = L(ŷ),  ŷ = f(z),  z = Wx + b
  dL/dW = (dL/dŷ) · (dŷ/dz) · (dz/dW)
  Each term computed by backprop, layer by layer

CODE

PYTHON
1import numpy as np
2from sympy import *
3
4x = symbols('x')
5
6# ── All differentiation rules demonstrated ─────────────────────────
7f = x**3
8g = sin(x)
9
10print("Sum rule:      d/dx(x³ + sin(x)) =", diff(f + g, x))
11print("Product rule:  d/dx(x³ · sin(x)) =", diff(f * g, x))
12print("Quotient rule: d/dx(x³ / sin(x)) =", simplify(diff(f / g, x)))
13print("Chain rule:    d/dx(sin(x³))     =", diff(sin(x**3), x))
14
15# ── Chain rule step by step ───────────────────────────────────────
16print("
17Chain rule: f(x) = log(sqrt(x))")
18inner = sqrt(x)           # g(x) = sqrt(x)
19outer = log(inner)        # f(g) = log(g)
20
21d_inner  = diff(inner, x)           # dg/dx = 1/(2√x)
22d_outer  = diff(log(symbols('g')), symbols('g'))  # df/dg = 1/g
23result   = simplify(diff(log(sqrt(x)), x))
24
25print(f"  dg/dx      = {d_inner}")
26print(f"  df/dg      = {d_outer}")
27print(f"  df/dx      = {result}")    # 1/(2x)
28
29# ── Sigmoid derivative (using chain rule) ─────────────────────────
30# sigma(x) = 1/(1+e^-x)
31# Let g = 1 + e^-x, f = 1/g = g^-1
32# df/dx = df/dg * dg/dx = -g^-2 * (-e^-x)
33#       = e^-x / (1+e^-x)^2
34#       = sigma(x) * (1 - sigma(x))  ← clean result!
35def sigmoid(x):   return 1 / (1 + np.exp(-x))
36def d_sigmoid(x): return sigmoid(x) * (1 - sigmoid(x))
37
38x_val = np.array([-2, -1, 0, 1, 2])
39print("
40Sigmoid and its derivative:")
41for xi in x_val:
42    print(f"  x={xi:+d}: σ={sigmoid(xi):.4f}, σ'={d_sigmoid(xi):.4f}")
43
44# ── Long chain rule (arbitrary depth) ────────────────────────────
45# f = sin(log(sqrt(x^2+1)))
46f_composed = sin(log(sqrt(x**2 + 1)))
47print(f"
48d/dx sin(log(sqrt(x²+1))) = {simplify(diff(f_composed, x))}")
← PREV2. Derivatives — IntuitionNEXT →4. Partial Derivatives & Gradients