Random Forest

Random Forest is an ensemble of decision trees trained on random subsets of data and features. Two sources of randomness:

1. Bootstrap sampling (Bagging) → each tree trained on ~63% of data, sampled with replacement
2. Feature subsampling → at each split, only √n_features are considered

Why this works — bias-variance tradeoff:
• Single deep tree: low bias, high variance (overfits)
• Average of many uncorrelated trees: low bias, lower variance

Because trees are decorrelated (different data + different features), their errors cancel out when averaged. This is the core insight.

Out-of-bag (OOB) score: samples not used in a tree's bootstrap sample can be used as validation — you get a free cross-validation estimate.

Random Forest is often the best "first serious model" to try on tabular data before gradient boosting.

1from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
2from sklearn.datasets import make_classification
3from sklearn.model_selection import train_test_split, cross_val_score
4from sklearn.metrics import accuracy_score, classification_report
5import numpy as np
6import matplotlib.pyplot as plt
7
8X, y = make_classification(n_samples=2000, n_features=20,
9                            n_informative=10, random_state=42)
10X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
11
12# ── Train Random Forest ───────────────────────────────────────────
13rf = RandomForestClassifier(
14    n_estimators=200,           # number of trees (more = better, diminishing returns)
15    max_depth=None,             # None = grow fully (trees are deep)
16    min_samples_split=5,
17    min_samples_leaf=2,
18    max_features="sqrt",        # √n_features at each split
19    bootstrap=True,             # bagging
20    oob_score=True,             # free OOB validation estimate
21    n_jobs=-1,                  # use all CPU cores
22    random_state=42,
23)
24rf.fit(X_train, y_train)
25
26print(f"Test accuracy : {accuracy_score(y_test, rf.predict(X_test)):.4f}")
27print(f"OOB score     : {rf.oob_score_:.4f}")   # no cross-val needed
28
29# ── Feature importance ────────────────────────────────────────────
30importances = rf.feature_importances_
31indices     = np.argsort(importances)[::-1][:10]   # top 10
32print("\nTop 10 features:")
33for i in indices:
34    print(f"  Feature {i:2d}: {importances[i]:.4f}")
35
36# ── n_estimators vs performance (find elbow) ─────────────────────
37scores = []
38for n in [10, 25, 50, 100, 200, 300, 500]:
39    r = RandomForestClassifier(n_estimators=n, n_jobs=-1, random_state=42)
40    r.fit(X_train, y_train)
41    scores.append(accuracy_score(y_test, r.predict(X_test)))
42    print(f"n={n:4d} trees → accuracy: {scores[-1]:.4f}")
43# Accuracy plateaus — 200-300 trees usually sufficient