MACHINE LEARNING / 5. GRADIENT BOOSTING & XGBOOST

Gradient Boosting & XGBoost

The best model on tabular data — period

EXPLANATION

Gradient Boosting builds trees sequentially, where each new tree corrects the errors of all previous trees. It literally trains on the residuals.

Algorithm:
1. Start with a simple prediction (mean of y)
2. Compute residuals = y - ŷ
3. Fit a new tree to predict the residuals
4. Update: ŷ = ŷ + lr * new_tree(X)
5. Repeat N times

XGBoost improvements over vanilla GBM:
• Regularization (L1/L2 on leaf weights) → prevents overfitting
• Second-order gradients (Newton boosting) → better optimization
• Column/row subsampling → like random forest, adds variance reduction
• Parallel tree building → much faster
• Handles missing values natively

LightGBM is even faster — uses histogram-based splits and grows trees leaf-wise rather than level-wise.

In practice: XGBoost/LightGBM win every Kaggle tabular competition. If you're on structured data, try these before anything else.

DATA FLOW

Iteration 1: predict mean(y). Residuals = y - mean(y)
       ↓
  Tree 1: fits residuals → small corrections
       ↓
  ŷ = mean(y) + lr × Tree1(X)
       ↓
  New residuals = y - ŷ
       ↓
  Tree 2: fits new residuals → more corrections
       ↓
  ŷ = mean(y) + lr×Tree1 + lr×Tree2
       ↓
  ... repeat N times ...
       ↓
  Final: ŷ = Σ lr × Treeᵢ(X)   ← ensemble of weak learners

CODE

PYTHON
1import xgboost as xgb
2import lightgbm as lgb
3from sklearn.datasets import make_classification
4from sklearn.model_selection import train_test_split, cross_val_score
5from sklearn.metrics import accuracy_score, roc_auc_score
6import numpy as np
7
8X, y = make_classification(n_samples=5000, n_features=20,
9                            n_informative=12, random_state=42)
10X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
11
12# ── XGBoost ───────────────────────────────────────────────────────
13xgb_model = xgb.XGBClassifier(
14    n_estimators=500,
15    max_depth=6,             # tree depth (3-8 typical)
16    learning_rate=0.05,      # shrinkage — lower = slower but better
17    subsample=0.8,           # row subsampling per tree
18    colsample_bytree=0.8,    # feature subsampling per tree
19    reg_alpha=0.1,           # L1 regularization
20    reg_lambda=1.0,          # L2 regularization
21    eval_metric="auc",
22    early_stopping_rounds=50,   # stop if no improvement for 50 rounds
23    random_state=42,
24)
25xgb_model.fit(
26    X_train, y_train,
27    eval_set=[(X_test, y_test)],
28    verbose=50,
29)
30print(f"XGB  AUC: {roc_auc_score(y_test, xgb_model.predict_proba(X_test)[:,1]):.4f}")
31
32# ── LightGBM (faster, often better) ──────────────────────────────
33lgb_model = lgb.LGBMClassifier(
34    n_estimators=500,
35    max_depth=-1,            # -1 = no limit
36    num_leaves=31,           # main complexity param for LGBM
37    learning_rate=0.05,
38    subsample=0.8,
39    colsample_bytree=0.8,
40    reg_alpha=0.1,
41    reg_lambda=1.0,
42    random_state=42,
43    verbose=-1,
44)
45lgb_model.fit(X_train, y_train,
46              eval_set=[(X_test, y_test)],
47              callbacks=[lgb.early_stopping(50, verbose=False)])
48print(f"LGBM AUC: {roc_auc_score(y_test, lgb_model.predict_proba(X_test)[:,1]):.4f}")
49
50# ── Feature importance ────────────────────────────────────────────
51importances = xgb_model.feature_importances_
52top_idx     = np.argsort(importances)[::-1][:5]
53for i in top_idx:
54    print(f"  Feature {i}: {importances[i]:.4f}")
← PREV4. Random ForestNEXT →6. SVM