MACHINE LEARNING / 1. LINEAR REGRESSION

Linear Regression

Predicting continuous values — the foundation of everything

EXPLANATION

Linear regression models the relationship between features X and a continuous target y as a straight line (or hyperplane in higher dimensions).

ŷ = w₁x₁ + w₂x₂ + ... + wₙxₙ + b = Xw + b

Training = finding w and b that minimize Mean Squared Error:
MSE = (1/n) Σ(yᵢ - ŷᵢ)²

Closed-form solution (Normal Equation): w = (XᵀX)⁻¹Xᵀy
Works perfectly for small datasets. For large datasets, use gradient descent.

Key assumptions:
• Linear relationship between X and y
• Features are independent (no multicollinearity)
• Residuals are normally distributed with constant variance

Regularization prevents overfitting:
• Ridge (L2) → penalizes large weights, shrinks all weights
• Lasso (L1) → drives some weights to exactly zero (feature selection)
• ElasticNet → combines both L1 and L2

DATA FLOW

Data points:          ●  ●
                    ●          ●
                ●       ●
            ●               ●

  Linear fit:   ─────────────────── ŷ = wx + b

  Residuals (errors):
  ●              ↕ (y - ŷ) for each point
  ───────────────────────────────────
  Goal: minimize sum of squared residuals

  Ridge adds: λΣwᵢ²   → shrinks weights
  Lasso adds: λΣ|wᵢ|  → zeroes out weights

CODE

PYTHON
1import numpy as np
2from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet
3from sklearn.model_selection import train_test_split
4from sklearn.preprocessing import StandardScaler
5from sklearn.metrics import mean_squared_error, r2_score
6import matplotlib.pyplot as plt
7
8# ── Generate toy dataset ─────────────────────────────────────────
9np.random.seed(42)
10X = np.random.randn(200, 3)
11y = 3*X[:,0] - 2*X[:,1] + 0.5*X[:,2] + np.random.randn(200)*0.5
12
13X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
14
15# Always scale features for linear models
16scaler  = StandardScaler()
17X_train = scaler.fit_transform(X_train)
18X_test  = scaler.transform(X_test)      # use train stats on test!
19
20# ── Ordinary Least Squares ────────────────────────────────────────
21lr = LinearRegression()
22lr.fit(X_train, y_train)
23y_pred = lr.predict(X_test)
24
25print(f"Coefficients : {lr.coef_}")
26print(f"Intercept    : {lr.intercept_:.4f}")
27print(f"R²           : {r2_score(y_test, y_pred):.4f}")
28print(f"RMSE         : {mean_squared_error(y_test, y_pred, squared=False):.4f}")
29
30# ── Ridge (L2 regularization) ────────────────────────────────────
31ridge = Ridge(alpha=1.0)    # alpha = λ, higher = more regularization
32ridge.fit(X_train, y_train)
33
34# ── Lasso (L1 regularization — feature selection) ────────────────
35lasso = Lasso(alpha=0.1)    # drives small weights to exactly 0
36lasso.fit(X_train, y_train)
37print(f"Lasso zeroed features: {(lasso.coef_ == 0).sum()}")
38
39# ── ElasticNet (L1 + L2) ─────────────────────────────────────────
40enet = ElasticNet(alpha=0.1, l1_ratio=0.5)  # l1_ratio: 0=Ridge, 1=Lasso
41enet.fit(X_train, y_train)
42
43# ── Compare all ──────────────────────────────────────────────────
44models = {"OLS": lr, "Ridge": ridge, "Lasso": lasso, "ElasticNet": enet}
45for name, model in models.items():
46    r2 = r2_score(y_test, model.predict(X_test))
47    print(f"{name:12s} R²: {r2:.4f}")
← PREVOverviewNEXT →2. Logistic Regression