Partial Derivatives & Gradients

Most ML functions take many inputs (all the weights of a network). Partial derivatives extend the concept of derivatives to multivariable functions.

Partial derivative ∂f/∂xᵢ: derivative of f with respect to xᵢ while treating all other variables as constants.

The gradient ∇f is a vector of all partial derivatives:
∇f(x₁,...,xₙ) = [∂f/∂x₁, ∂f/∂x₂, ..., ∂f/∂xₙ]ᵀ

The gradient points in the direction of steepest ascent.
Negative gradient points toward steepest descent — that's gradient descent!

Gradient descent update rule:
θ = θ - lr · ∇_θ L(θ)

For a neural network with millions of parameters, the gradient is a vector with millions of entries — one per parameter. PyTorch computes all of these with a single call to loss.backward() using the chain rule recursively.

From Raschka's notation:
∂f/∂x means "differentiate f with respect to x, hold everything else constant."
The symbol ∂ (curly d) distinguishes partial from ordinary derivatives.

1import numpy as np
2from sympy import *
3import torch
4
5# ── Symbolic partial derivatives ──────────────────────────────────
6x, y = symbols('x y')
7
8f = x**2 * y + y    # Raschka's example
9
10df_dx = diff(f, x)   # partial w.r.t. x
11df_dy = diff(f, y)   # partial w.r.t. y
12
13print(f"f(x,y)  = {f}")
14print(f"∂f/∂x   = {df_dx}")    # 2xy
15print(f"∂f/∂y   = {df_dy}")    # x^2 + 1
16print(f"∇f      = [{df_dx}, {df_dy}]")
17
18# At point (2, 3)
19point = {x: 2, y: 3}
20print(f"
21At (2,3):")
22print(f"  ∂f/∂x = {df_dx.subs(point)}")   # 12
23print(f"  ∂f/∂y = {df_dy.subs(point)}")   # 5
24
25# ── PyTorch autograd — how real ML computes gradients ─────────────
26# PyTorch builds a computation graph and applies chain rule automatically
27import torch
28
29# Loss = (w1*x + w2*x^2 - y)^2  — toy regression
30w1 = torch.tensor(1.0, requires_grad=True)
31w2 = torch.tensor(1.0, requires_grad=True)
32
33x_data = torch.tensor(2.0)
34y_data = torch.tensor(5.0)
35
36prediction = w1 * x_data + w2 * x_data**2
37loss       = (prediction - y_data)**2
38
39loss.backward()   # computes all gradients via chain rule
40
41print(f"
42PyTorch autograd:")
43print(f"  prediction = {prediction.item():.2f}")
44print(f"  loss       = {loss.item():.2f}")
45print(f"  ∂L/∂w1     = {w1.grad.item():.4f}")
46print(f"  ∂L/∂w2     = {w2.grad.item():.4f}")
47
48# ── Gradient descent on 2D surface ───────────────────────────────
49def f_np(x, y):   return x**2 + 2*y**2 + x*y   # bowl-shaped
50
51def grad_f(x, y):
52    df_dx = 2*x + y     # ∂f/∂x
53    df_dy = 4*y + x     # ∂f/∂y
54    return np.array([df_dx, df_dy])
55
56theta = np.array([4.0, 3.0])   # start far from minimum
57lr = 0.1
58
59print("
60Gradient descent on f(x,y) = x²+2y²+xy:")
61for step in range(10):
62    g = grad_f(*theta)
63    theta = theta - lr * g
64    print(f"  step {step+1:2d}: θ={theta}, f={f_np(*theta):.4f}")