LINEAR ALGEBRA / 6. EIGENVALUES & EIGENVECTORS

Eigenvalues & Eigenvectors

3Blue1Brown: vectors that only get stretched, never rotated

EXPLANATION

3Blue1Brown's definition: eigenvectors are the special vectors that stay on their own span after a transformation — they only get scaled, not rotated.

Av = λv
• v → eigenvector (the direction that doesn't rotate)
• λ → eigenvalue (how much v gets scaled)

To find: det(A - λI) = 0  (characteristic equation)
Then solve (A - λI)v = 0 for each λ.

Geometric intuition:
• λ > 1 → stretch in this direction
• 0 < λ < 1 → compression
• λ < 0 → flip and scale
• λ = 0 → collapse to zero (singular matrix!)

Why eigenvectors matter for ML:
• PCA: eigenvectors of covariance matrix = principal components (directions of max variance)
• Hessian: eigenvectors = principal curvature directions, eigenvalues = curvature magnitude
• Graph neural networks: graph Laplacian eigenvectors
• Markov chains: stationary distribution = eigenvector with λ=1
• Attention: implicitly related to eigendecomposition of attention matrix

Symmetric matrices (like covariance matrices, Hessians):
• Always have real eigenvalues
• Eigenvectors are orthogonal to each other
• Can be eigendecomposed as A = QΛQᵀ

DIAGRAM

A = [[3, 1],
       [0, 2]]

  Characteristic equation: det(A - λI) = 0
  det([[3-λ, 1],
       [0,  2-λ]]) = (3-λ)(2-λ) = 0

  Eigenvalues: λ₁=3, λ₂=2

  For λ₁=3: (A-3I)v=0 → v₁=[1,0] (x-axis unchanged!)
  For λ₂=2: (A-2I)v=0 → v₂=[1,-1]

  Geometric: x-axis stretches by 3, v₂ direction stretches by 2
  All other vectors rotate AND scale

  PCA connection:
  Covariance Σ → eigenvectors = principal directions
                 eigenvalues  = variance in each direction

CODE

PYTHON
1import numpy as np
2import matplotlib.pyplot as plt
3
4# ── Computing eigenvalues and eigenvectors ────────────────────────
5A = np.array([[3., 1.], [0., 2.]])
6eigenvalues, eigenvectors = np.linalg.eig(A)
7
8print("Eigendecomposition:")
9print(f"  Eigenvalues:  {eigenvalues}")
10print(f"  Eigenvectors:
11{eigenvectors}")
12
13# Verify: Av = λv
14for i, (lam, vec) in enumerate(zip(eigenvalues, eigenvectors.T)):
15    Av  = A @ vec
16    lv  = lam * vec
17    print(f"
18  λ={lam}: Av={Av.round(4)}, λv={lv.round(4)}, match={np.allclose(Av,lv)}")
19
20# ── Symmetric matrix — real eigenvalues, orthogonal eigenvectors ──
21S = np.array([[4., 2.], [2., 3.]])   # symmetric
22vals, vecs = np.linalg.eigh(S)       # eigh for symmetric (more stable)
23print(f"
24Symmetric matrix eigenvalues: {vals}")
25print(f"Eigenvectors orthogonal: {np.isclose(vecs[:,0] @ vecs[:,1], 0)}")
26
27# Eigendecomposition: S = Q Λ Qᵀ
28Q   = vecs
29Lam = np.diag(vals)
30S_reconstructed = Q @ Lam @ Q.T
31print(f"S = QΛQᵀ reconstruction error: {np.max(abs(S - S_reconstructed)):.2e}")
32
33# ── PCA from scratch using eigendecomposition ─────────────────────
34np.random.seed(42)
35# Generate correlated 2D data
36mean = [0, 0]
37cov  = [[3, 2], [2, 2]]
38data = np.random.multivariate_normal(mean, cov, 200)
39
40# Center
41data_centered = data - data.mean(axis=0)
42
43# Covariance matrix
44Sigma = np.cov(data_centered.T)
45print(f"
46Covariance matrix:
47{Sigma.round(3)}")
48
49# Eigendecomposition of covariance
50eig_vals, eig_vecs = np.linalg.eigh(Sigma)
51# Sort by eigenvalue descending
52idx      = np.argsort(eig_vals)[::-1]
53eig_vals = eig_vals[idx]
54eig_vecs = eig_vecs[:, idx]
55
56print(f"
57PCA via eigendecomposition:")
58print(f"  PC1 explains {eig_vals[0]/eig_vals.sum()*100:.1f}% variance")
59print(f"  PC2 explains {eig_vals[1]/eig_vals.sum()*100:.1f}% variance")
60print(f"  PC1 direction: {eig_vecs[:,0].round(3)}")
61
62# Project data onto principal components
63data_pca = data_centered @ eig_vecs
64print(f"  Data projected shape: {data_pca.shape}")
← PREV5. Linear SystemsNEXT →7. Dot Product & Norms