Derivatives — Intuition & Definition

The derivative measures how fast a function changes at a specific point.

Formal definition (from Sebastian Raschka's appendix):
f'(x) = df/dx = lim(Δx→0) [f(x+Δx) - f(x)] / Δx

Geometric interpretation: the slope of the tangent line to the curve at point x.

Two notations (both mean the same thing):
• Lagrange: f'(x)   ← prime notation
• Leibniz:  df/dx   ← fraction notation, reads as "d of f with respect to x"

Leibniz notation is preferred in ML because it makes the chain rule intuitive: df/dx = (df/dg) · (dg/dx) — the dg terms "cancel" visually.

Key derivatives to memorize:
• d/dx(xⁿ) = n·xⁿ⁻¹         Power rule
• d/dx(eˣ) = eˣ               Exponential
• d/dx(log x) = 1/x           Natural log
• d/dx(sin x) = cos x
• d/dx(cos x) = -sin x

Sign of derivative tells you everything:
• f'(x) > 0 → function increasing at x
• f'(x) < 0 → function decreasing at x
• f'(x) = 0 → critical point (potential min/max)

1import numpy as np
2from sympy import *
3import matplotlib.pyplot as plt
4
5x = symbols('x')
6
7# ── Symbolic derivatives ──────────────────────────────────────────
8functions = {
9    "x^3":          x**3,
10    "e^x":          exp(x),
11    "log(x)":       log(x),
12    "sin(x)":       sin(x),
13    "x^2 * log(x)": x**2 * log(x),
14    "1/(1+e^-x)":   1/(1+exp(-x)),  # sigmoid!
15}
16
17print("Derivatives:")
18for name, f in functions.items():
19    print(f"  d/dx({name:15s}) = {diff(f, x)}")
20
21# ── Sigmoid derivative (important for backprop) ───────────────────
22sigma = 1 / (1 + exp(-x))
23d_sigma = diff(sigma, x)
24d_sigma_simplified = simplify(d_sigma)
25print(f"
26Sigmoid derivative: {d_sigma_simplified}")
27# = sigma * (1 - sigma) ← elegant!
28
29# ── Numerical derivatives — useful for debugging ──────────────────
30def numerical_grad(f, x_val, eps=1e-5):
31    """Central difference — more accurate than forward difference."""
32    return (f(x_val + eps) - f(x_val - eps)) / (2 * eps)
33
34f_test = lambda x: np.log(x**2 + 1) * np.sin(x)
35x0     = 2.0
36
37num_grad  = numerical_grad(f_test, x0)
38# Analytical using sympy
39f_sym     = log(x**2 + 1) * sin(x)
40ana_grad  = float(diff(f_sym, x).subs(x, x0))
41
42print(f"
43At x={x0}:")
44print(f"  Numerical gradient  : {num_grad:.8f}")
45print(f"  Analytical gradient : {ana_grad:.8f}")
46print(f"  Match: {np.isclose(num_grad, ana_grad, rtol=1e-5)}")
47
48# ── Visualize: function vs its derivative ─────────────────────────
49x_vals = np.linspace(-3, 3, 300)
50f_vals  = x_vals**3 - 2*x_vals
51df_vals = 3*x_vals**2 - 2
52
53plt.figure(figsize=(10, 4))
54plt.subplot(1, 2, 1)
55plt.plot(x_vals, f_vals, color="#FB923C", linewidth=2, label="f(x) = x³-2x")
56plt.axhline(0, color="gray", linewidth=0.5)
57plt.title("Function"); plt.legend()
58
59plt.subplot(1, 2, 2)
60plt.plot(x_vals, df_vals, color="#FDBA74", linewidth=2, label="f'(x) = 3x²-2")
61plt.axhline(0, color="gray", linewidth=0.5)
62plt.title("Derivative"); plt.legend()
63plt.tight_layout()
64plt.savefig("derivative_demo.png", dpi=150)