DEEP LEARNING / 3. OPTIMIZERS

Optimizers

How we update weights — from vanilla SGD to Adam

EXPLANATION

Gradient descent says: move weights in the direction that reduces loss. But naive gradient descent has problems — slow convergence, stuck in saddle points, different learning rates needed per parameter.

Key optimizers:
• SGD           → w = w - lr * grad. Simple, noisy, needs careful lr tuning
• SGD + Momentum → accumulates velocity, smooths updates, escapes local minima
• RMSProp       → adapts lr per parameter using running average of squared gradients
• Adam          → combines Momentum + RMSProp. The default choice for most tasks
• AdamW         → Adam + proper weight decay. Best for transformers

In practice: start with Adam or AdamW. Use SGD + momentum for CNNs on vision tasks where it often edges out Adam on final accuracy.

DATA FLOW

Loss surface (imagine a hilly landscape):

  SGD:          bounces around, slow, noisy
  SGD+Momentum: builds speed downhill, overshoots less
  Adam:         adapts step size per weight automatically

  Adam update rule:
  m = β1·m + (1-β1)·grad        ← momentum (1st moment)
  v = β2·v + (1-β2)·grad²       ← RMS (2nd moment)
  m̂ = m / (1-β1^t)              ← bias correction
  v̂ = v / (1-β2^t)              ← bias correction
  w = w - lr · m̂ / (√v̂ + ε)    ← adaptive update

CODE

PYTHON
1import torch
2import torch.nn as nn
3import torch.optim as optim
4
5model = nn.Linear(10, 1)
6
7# ── All major optimizers ──────────────────────────────────────────
8sgd       = optim.SGD(model.parameters(), lr=0.01)
9sgd_mom   = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
10rmsprop   = optim.RMSprop(model.parameters(), lr=0.001, alpha=0.99)
11adam      = optim.Adam(model.parameters(), lr=1e-3, betas=(0.9, 0.999))
12adamw     = optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)
13
14# ── Learning rate schedulers (just as important as optimizer) ──────
15optimizer = optim.AdamW(model.parameters(), lr=1e-3)
16
17# StepLR: reduce lr by gamma every step_size epochs
18step_sched = optim.lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1)
19
20# CosineAnnealingLR: smoothly decays lr following cosine curve
21cosine_sched = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=100)
22
23# OneCycleLR: warmup then decay — best for training from scratch
24onecycle = optim.lr_scheduler.OneCycleLR(
25    optimizer, max_lr=1e-2, steps_per_epoch=100, epochs=10
26)
27
28# ── Compare optimizers on same task ──────────────────────────────
29def train(optimizer_name):
30    torch.manual_seed(42)
31    model = nn.Sequential(nn.Linear(10, 64), nn.ReLU(), nn.Linear(64, 1))
32    X, y  = torch.randn(200, 10), torch.randn(200, 1)
33    loss_fn = nn.MSELoss()
34
35    opts = {
36        "SGD":   optim.SGD(model.parameters(), lr=0.01),
37        "Adam":  optim.Adam(model.parameters(), lr=1e-3),
38        "AdamW": optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01),
39    }
40    opt = opts[optimizer_name]
41
42    for _ in range(200):
43        opt.zero_grad()
44        loss_fn(model(X), y).backward()
45        opt.step()
46
47    return loss_fn(model(X), y).item()
48
49for name in ["SGD", "Adam", "AdamW"]:
50    print(f"{name:6s} final loss: {train(name):.6f}")
← PREV2. BackpropagationNEXT →4. CNNs