MACHINE LEARNING / 8. PCA & DIMENSIONALITY REDUCTION

PCA & Dimensionality Reduction

Compressing data while preserving variance

EXPLANATION

High-dimensional data has the "curse of dimensionality" — distances become meaningless, models overfit, visualization is impossible. Dimensionality reduction fixes this.

PCA (Principal Component Analysis):
• Finds orthogonal directions (principal components) of maximum variance
• Projects data onto top-K components
• Linear transformation — new features are combinations of original features
• Unsupervised — doesn't use labels

Algorithm:
1. Center the data (subtract mean)
2. Compute covariance matrix
3. Compute eigenvectors and eigenvalues
4. Sort by eigenvalue (variance explained)
5. Project onto top-K eigenvectors

t-SNE and UMAP: non-linear, used for visualization only (2D/3D). t-SNE preserves local structure, UMAP is faster and preserves more global structure.

Rule of thumb: use PCA before training if you have 100+ features, or as preprocessing to remove noise. Use t-SNE/UMAP for visualization.

DATA FLOW

Original data: 100 features
        ↓
  PCA finds directions of maximum variance:
  PC1: direction explaining most variance (say 40%)
  PC2: orthogonal to PC1, explains next most (25%)
  PC3: orthogonal to both, explains next (15%)
  ...
        ↓
  Keep top K components (explaining 95% variance)
        ↓
  Reduced data: K features (e.g. K=10 instead of 100)

  Explained variance ratio:
  PC1: ████████████████ 40%
  PC2: ██████████ 25%
  PC3: ██████ 15%
  ...  sum to 95% → keep these

CODE

PYTHON
1import numpy as np
2from sklearn.decomposition import PCA
3from sklearn.manifold import TSNE
4from sklearn.preprocessing import StandardScaler
5from sklearn.datasets import load_digits
6import matplotlib.pyplot as plt
7
8# ── Load high-dimensional dataset (8×8 images = 64 features) ─────
9digits = load_digits()
10X      = digits.data      # (1797, 64)
11y      = digits.target
12
13X = StandardScaler().fit_transform(X)   # always scale before PCA
14
15# ── PCA ───────────────────────────────────────────────────────────
16pca = PCA(n_components=0.95)    # keep components explaining 95% variance
17X_pca = pca.fit_transform(X)
18
19print(f"Original shape  : {X.shape}")       # (1797, 64)
20print(f"Reduced shape   : {X_pca.shape}")   # (1797, ~29)
21print(f"Components kept : {pca.n_components_}")
22print(f"Variance explained: {pca.explained_variance_ratio_.sum():.4f}")
23
24# ── Scree plot ────────────────────────────────────────────────────
25pca_full = PCA()
26pca_full.fit(X)
27cumvar = np.cumsum(pca_full.explained_variance_ratio_)
28# plt.plot(cumvar) → shows elbow around 10-15 components
29
30# ── Use PCA as preprocessing → then classify ─────────────────────
31from sklearn.linear_model import LogisticRegression
32from sklearn.pipeline import Pipeline
33from sklearn.model_selection import cross_val_score
34
35pipe_raw = Pipeline([("lr", LogisticRegression(max_iter=1000))])
36pipe_pca = Pipeline([
37    ("pca", PCA(n_components=20)),
38    ("lr", LogisticRegression(max_iter=1000)),
39])
40print(f"Raw features CV : {cross_val_score(pipe_raw, X, y, cv=5).mean():.4f}")
41print(f"PCA features CV : {cross_val_score(pipe_pca, X, y, cv=5).mean():.4f}")
42
43# ── t-SNE for 2D visualization ────────────────────────────────────
44tsne    = TSNE(n_components=2, perplexity=30, random_state=42)
45X_tsne  = tsne.fit_transform(X[:500])       # slow — use subset
46y_sub   = y[:500]
47
48plt.figure(figsize=(8, 6))
49scatter = plt.scatter(X_tsne[:,0], X_tsne[:,1], c=y_sub,
50                      cmap="tab10", alpha=0.7, s=15)
51plt.colorbar(scatter)
52plt.title("t-SNE of Digits Dataset")
53plt.savefig("tsne.png", dpi=150)
← PREV7. ClusteringNEXT →9. Evaluation & Pipelines