MACHINE LEARNING / 2. LOGISTIC REGRESSION

Logistic Regression

Classification via probability — not actually regression

EXPLANATION

Despite the name, logistic regression is a classification algorithm. It predicts the probability that a sample belongs to a class.

The key: wrap linear regression output in a sigmoid function to squash it to (0, 1):
p = σ(Xw + b) = 1 / (1 + e^-(Xw+b))

Training minimizes Binary Cross-Entropy (Log Loss):
L = -[y·log(p) + (1-y)·log(1-p)]

Decision boundary: predict class 1 if p >= 0.5, else class 0.

For multi-class:
• One-vs-Rest (OvR) → train N binary classifiers
• Softmax (multinomial) → generalize sigmoid to K classes

Why logistic regression is still relevant:
• Interpretable (coefficients = feature importances)
• Fast to train and predict
• Works well on linearly separable data
• Great baseline before trying complex models
• Used in ad click prediction at massive scale

DATA FLOW

Linear output z = Xw + b   (any real number)
          ↓
  Sigmoid: p = 1/(1+e^-z)    (squashed to 0-1)
          ↓
  Decision: class = 1 if p >= 0.5

  Sigmoid curve:
  1.0 |          ──────────
      |       ──/
  0.5 |     ──/
      |  ──/
  0.0 |──
      └──────────────────── z
              0

  Log loss penalizes confident wrong predictions heavily

CODE

PYTHON
1import numpy as np
2from sklearn.linear_model import LogisticRegression
3from sklearn.datasets import make_classification
4from sklearn.model_selection import train_test_split, cross_val_score
5from sklearn.preprocessing import StandardScaler
6from sklearn.metrics import (classification_report, confusion_matrix,
7                              roc_auc_score, roc_curve)
8import matplotlib.pyplot as plt
9
10# ── Dataset ───────────────────────────────────────────────────────
11X, y = make_classification(n_samples=1000, n_features=10,
12                            n_informative=6, random_state=42)
13X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
14
15scaler  = StandardScaler()
16X_train = scaler.fit_transform(X_train)
17X_test  = scaler.transform(X_test)
18
19# ── Train logistic regression ─────────────────────────────────────
20model = LogisticRegression(
21    C=1.0,                  # C = 1/λ (inverse regularization strength)
22    penalty="l2",           # "l1", "l2", "elasticnet", None
23    solver="lbfgs",         # "lbfgs", "liblinear", "saga"
24    max_iter=1000,
25    random_state=42,
26)
27model.fit(X_train, y_train)
28
29# ── Predict ───────────────────────────────────────────────────────
30y_pred  = model.predict(X_test)
31y_proba = model.predict_proba(X_test)[:, 1]  # probability of class 1
32
33print(classification_report(y_test, y_pred))
34
35# ── AUC-ROC ───────────────────────────────────────────────────────
36auc = roc_auc_score(y_test, y_proba)
37print(f"AUC-ROC: {auc:.4f}")   # 1.0 = perfect, 0.5 = random
38
39fpr, tpr, _ = roc_curve(y_test, y_proba)
40plt.plot(fpr, tpr, label=f"AUC = {auc:.3f}", color="#FF6B6B")
41plt.plot([0,1],[0,1], "k--", alpha=0.3)
42plt.xlabel("False Positive Rate")
43plt.ylabel("True Positive Rate")
44plt.title("ROC Curve")
45plt.legend()
46plt.savefig("roc.png", dpi=150)
47
48# ── Cross validation ──────────────────────────────────────────────
49from sklearn.pipeline import Pipeline
50pipe = Pipeline([("scaler", StandardScaler()), ("lr", LogisticRegression())])
51cv_scores = cross_val_score(pipe, X, y, cv=5, scoring="roc_auc")
52print(f"CV AUC: {cv_scores.mean():.4f} ± {cv_scores.std():.4f}")
← PREV1. Linear RegressionNEXT →3. Decision Trees