CALCULUS / 6. SECOND ORDER & HESSIAN

Second Order Derivatives & The Hessian

Curvature — how fast the gradient is changing

EXPLANATION

The second derivative measures the rate of change of the derivative — the curvature of the function.

d²f/dx² > 0 → concave up (bowl) → local minimum below
d²f/dx² < 0 → concave down (hill) → local maximum below
d²f/dx² = 0 → inflection point

For multivariable functions, the second-order information is captured by the Hessian matrix H.

From Raschka's appendix, for f: Rⁿ → R:
Hf[i,j] = ∂²f / (∂xᵢ ∂xⱼ)

The Hessian is an n×n matrix of all second-order partial derivatives.

Why it matters for ML:
• Positive definite Hessian → at a critical point (∇f=0), it's a minimum
• Newton's method uses H⁻¹∇f for faster optimization (but H⁻¹ is expensive for big networks)
• Condition number of H tells you how hard the optimization landscape is
• Adam optimizer approximates second-order information without computing H

The Laplacian Δf = Σ ∂²f/∂xᵢ² is the trace of the Hessian — sum of all second order partial derivatives w.r.t. the same variable.

DIAGRAM

f(x) = x³ - 3x:
  f'(x)  = 3x² - 3    (first derivative)
  f''(x) = 6x          (second derivative)

  At x=-1: f'=0 (critical), f''=-6 < 0 → LOCAL MAX
  At x=+1: f'=0 (critical), f''>+6 > 0 → LOCAL MIN

  Hessian of f(x,y) = x²y + y:
       ∂²f/∂x²  ∂²f/∂x∂y     2y  2x
  H = [                   ] = [       ]
       ∂²f/∂y∂x ∂²f/∂y²      2x   0

  At (1,1): H = [[2,2],[2,0]]
  det(H) = 0-4 = -4 < 0 → SADDLE POINT
  (neither min nor max — common in neural networks!)

CODE

PYTHON
1import numpy as np
2from sympy import *
3import matplotlib.pyplot as plt
4
5x, y = symbols('x y')
6
7# ── Second derivative: identify critical points ───────────────────
8f = x**3 - 3*x
9df   = diff(f, x)
10d2f  = diff(f, x, 2)    # second derivative
11
12print(f"f(x)   = {f}")
13print(f"f'(x)  = {df}")
14print(f"f''(x) = {d2f}")
15
16critical_points = solve(df, x)
17print(f"
18Critical points: {critical_points}")
19for cp in critical_points:
20    val = d2f.subs(x, cp)
21    kind = "minimum" if val > 0 else "maximum" if val < 0 else "inflection"
22    print(f"  x={cp}: f''={val} → {kind}")
23
24# ── Hessian matrix ─────────────────────────────────────────────────
25print("
26Hessian of f(x,y) = x²y + y:")
27f2 = x**2 * y + y
28
29H = hessian(f2, (x, y))
30print(f"  H = {H}")
31print(f"  At (1,1): H = {H.subs([(x,1),(y,1)])}")
32print(f"  det(H) at (1,1) = {det(H.subs([(x,1),(y,1)]))}")
33
34# ── Numerical Hessian ──────────────────────────────────────────────
35def numerical_hessian(f, x_vec, eps=1e-4):
36    """Compute Hessian numerically using finite differences."""
37    n = len(x_vec)
38    H = np.zeros((n, n))
39    for i in range(n):
40        for j in range(n):
41            xi_p = x_vec.copy(); xi_p[i] += eps; xi_p[j] += eps
42            xi_n = x_vec.copy(); xi_n[i] -= eps; xi_n[j] -= eps
43            xp_n = x_vec.copy(); xp_n[i] += eps; xp_n[j] -= eps
44            xn_p = x_vec.copy(); xn_p[i] -= eps; xn_p[j] += eps
45            H[i,j] = (f(xi_p) - f(xp_n) - f(xn_p) + f(xi_n)) / (4*eps**2)
46    return H
47
48f_num = lambda v: v[0]**2 * v[1] + v[1]
49H_num = numerical_hessian(f_num, np.array([1.0, 1.0]))
50print(f"
51Numerical Hessian at (1,1):
52{H_num}")
53
54# Eigenvalues of Hessian tell you the shape of loss surface
55eigenvalues = np.linalg.eigvalsh(H_num)
56print(f"Eigenvalues: {eigenvalues}")
57print(f"{'Saddle point' if any(eigenvalues<0) else 'Minimum'}")
← PREV5. Chain Rule & BackpropNEXT →7. Multivariable Chain Rule