Support Vector Machines

SVM finds the hyperplane that maximally separates classes. Instead of just finding any boundary, it finds the one with the largest margin — the distance to the nearest points from each class (support vectors).

Hard margin SVM: assumes data is linearly separable. Rarely holds in practice.
Soft margin SVM: allows some misclassifications (controlled by C):
• High C → small margin, fewer errors on training data (overfit risk)
• Low C  → large margin, allows more training errors (underfit risk)

Kernel trick: implicitly maps data to high-dimensional space where linear separation becomes possible, without computing the transformation explicitly.

Common kernels:
• Linear → dot product. Use for high-dimensional data (text)
• RBF (Radial Basis Function) → most common, handles non-linear boundaries
• Polynomial → for polynomial decision boundaries

SVM works great for: text classification, image classification (before deep learning), small-to-medium datasets with clear margins.

1from sklearn.svm import SVC, SVR
2from sklearn.datasets import make_classification, make_moons
3from sklearn.model_selection import train_test_split, GridSearchCV
4from sklearn.preprocessing import StandardScaler
5from sklearn.metrics import accuracy_score, classification_report
6import numpy as np
7
8# ── SVM is sensitive to scale — ALWAYS scale ─────────────────────
9X, y = make_moons(n_samples=1000, noise=0.2, random_state=42)
10X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
11
12scaler  = StandardScaler()
13X_train = scaler.fit_transform(X_train)
14X_test  = scaler.transform(X_test)
15
16# ── RBF kernel SVM ────────────────────────────────────────────────
17svm = SVC(
18    kernel="rbf",       # "linear", "rbf", "poly", "sigmoid"
19    C=1.0,              # regularization (higher = less regularized)
20    gamma="scale",      # RBF bandwidth. "scale" = 1/(n_features * X.var())
21    probability=True,   # enable predict_proba (slower)
22    random_state=42,
23)
24svm.fit(X_train, y_train)
25print(f"RBF SVM accuracy: {accuracy_score(y_test, svm.predict(X_test)):.4f}")
26print(f"Support vectors : {svm.n_support_}")   # per class
27
28# ── Kernel comparison ─────────────────────────────────────────────
29for kernel in ["linear", "rbf", "poly"]:
30    m = SVC(kernel=kernel, C=1.0, gamma="scale")
31    m.fit(X_train, y_train)
32    acc = accuracy_score(y_test, m.predict(X_test))
33    print(f"Kernel {kernel:7s}: {acc:.4f}")
34
35# ── Hyperparameter search (C and gamma) ───────────────────────────
36param_grid = {
37    "C":     [0.1, 1, 10, 100],
38    "gamma": [1, 0.1, 0.01, 0.001],
39}
40grid = GridSearchCV(SVC(kernel="rbf"), param_grid, cv=5,
41                    scoring="accuracy", n_jobs=-1)
42grid.fit(X_train, y_train)
43print(f"\nBest params : {grid.best_params_}")
44print(f"Best CV acc : {grid.best_score_:.4f}")