Integration

Integration is the inverse of differentiation (Fundamental Theorem of Calculus) and also computes the area under a curve.

Indefinite integral: ∫f(x)dx = F(x) + C where F'(x) = f(x)
Definite integral: ∫[a,b]f(x)dx = F(b) - F(a) = area from a to b

Fundamental Theorem of Calculus:
If F'(x) = f(x), then ∫[a,b]f(x)dx = F(b) - F(a)

Differentiation and integration are inverse operations.

Why integration matters for ML:
• Probability: P(a ≤ X ≤ b) = ∫[a,b]f(x)dx — the PDF integrates to probability
• Expected value: E[X] = ∫x·f(x)dx
• Normalization: ∫[−∞,+∞]f(x)dx = 1 for any valid PDF
• KL divergence, entropy, and many other ML quantities involve integrals
• Gaussian integrals appear constantly in probabilistic ML

Key integrals to know:
• ∫xⁿdx = xⁿ⁺¹/(n+1) + C
• ∫eˣdx = eˣ + C
• ∫(1/x)dx = log|x| + C
• ∫[−∞,+∞]e^(−x²)dx = √π  (Gaussian integral)

1import numpy as np
2from scipy import integrate
3from sympy import *
4import matplotlib.pyplot as plt
5
6x = symbols('x')
7
8# ── Symbolic integration ──────────────────────────────────────────
9print("Indefinite integrals:")
10print(f"  ∫x²dx   = {integrate(x**2, x)}")         # x³/3
11print(f"  ∫e^x dx = {integrate(exp(x), x)}")        # e^x
12print(f"  ∫1/x dx = {integrate(1/x, x)}")           # log(x)
13print(f"  ∫sin(x) = {integrate(sin(x), x)}")        # -cos(x)
14
15print("
16Definite integrals:")
17print(f"  ∫[0,3] x² dx = {integrate(x**2, (x, 0, 3))}")   # 9
18print(f"  ∫[0,1] e^x   = {integrate(exp(x), (x, 0, 1))}")  # e-1
19
20# ── Gaussian integral ─────────────────────────────────────────────
21# ∫[-∞,+∞] e^(-x²) dx = √π
22gaussian = integrate(exp(-x**2), (x, -oo, oo))
23print(f"
24  ∫[-∞,∞] e^(-x²) dx = {gaussian}")   # sqrt(pi)
25
26# ── Probability via integration ───────────────────────────────────
27# Standard normal PDF: f(x) = (1/√2π) e^(-x²/2)
28mu, sigma = 0, 1
29pdf = (1/sqrt(2*pi*sigma**2)) * exp(-(x-mu)**2 / (2*sigma**2))
30
31# P(-1 ≤ X ≤ 1) for standard normal
32prob_1sigma = integrate(pdf, (x, -1, 1))
33print(f"
34Normal distribution probabilities:")
35print(f"  P(-1 ≤ X ≤ 1) = {float(prob_1sigma):.4f}")  # ~0.6827
36
37# Numerical integration (when no closed form exists)
38from scipy.stats import norm
39prob_numerical, _ = integrate.quad(lambda x: norm.pdf(x), -1, 1)
40print(f"  Numerical:       {prob_numerical:.4f}")
41
42# ── Expected value via integration ───────────────────────────────
43# E[X²] for standard normal = variance = 1
44ex2 = float(integrate(x**2 * pdf, (x, -oo, oo)))
45print(f"
46E[X²] for standard normal = {ex2:.4f}")  # 1.0 (variance)
47
48# ── KL divergence (requires integration) ─────────────────────────
49# KL(P||Q) = ∫ p(x) log(p(x)/q(x)) dx
50# For two Gaussians N(μ₁,σ₁) and N(μ₂,σ₂):
51def kl_gaussians(mu1, s1, mu2, s2):
52    return np.log(s2/s1) + (s1**2 + (mu1-mu2)**2)/(2*s2**2) - 0.5
53
54print(f"
55KL(N(0,1) || N(1,1)) = {kl_gaussians(0,1,1,1):.4f}")
56print(f"KL(N(0,1) || N(0,2)) = {kl_gaussians(0,1,0,2):.4f}")