CALCULUS / 9. OPTIMIZATION
Optimization — Gradient Descent & Beyond
Finding minima — where calculus meets machine learning
EXPLANATION
Optimization is the goal of training: find the parameters θ that minimize the loss function L(θ). Critical points where ∇L = 0: • Local minimum: ∇L=0, Hessian positive definite • Local maximum: ∇L=0, Hessian negative definite • Saddle point: ∇L=0, Hessian indefinite (some eigenvalues +, some -) Deep neural networks have mostly saddle points, not local minima. This is actually OK — saddle points in high dimensions are usually easy to escape because there's always a descent direction. Gradient Descent variants: • Batch GD: use all data → exact gradient, slow • SGD: use one sample → noisy but fast • Mini-batch: use batch of 32-256 → best of both Learning rate is the most important hyperparameter: • Too large → overshoots, diverges • Too small → too slow, gets stuck • Adaptive methods (Adam) → adjust lr per parameter automatically Adam = Momentum + RMSProp: m = β₁m + (1-β₁)∇L (first moment — momentum) v = β₂v + (1-β₂)(∇L)² (second moment — variance) θ = θ - lr · m̂/√(v̂+ε) (adaptive update)
DIAGRAM
Loss surface — 2D visualization:
high loss
╲ saddle ╱
╲ × ╱
╲ ╱
min × × min
└─────┘
low loss (valley)
Gradient descent paths:
Large lr: zig-zags → overshoots valley
Small lr: slow descent → takes forever
Adam: adapts per-dimension → follows valley efficiently
Vanishing vs exploding gradients:
||∇L|| → 0 (too small → no learning)
||∇L|| → ∞ (too large → diverges)
Gradient clipping: clip ||∇L|| to max_normCODE