Evaluation & ML Pipelines

Choosing the right evaluation metric is as important as choosing the right model. Using the wrong metric gives you a false sense of performance.

Classification metrics:
• Accuracy   → correct / total. Misleading for imbalanced data
• Precision  → TP / (TP + FP). How many predicted positives are actually positive
• Recall     → TP / (TP + FN). How many actual positives did we catch
• F1         → harmonic mean of precision and recall. Balances both
• AUC-ROC    → area under ROC curve. Threshold-independent. Best for ranking

Regression metrics:
• MAE   → mean absolute error. Robust to outliers
• RMSE  → root mean squared error. Penalizes large errors more
• R²    → proportion of variance explained. 1.0 = perfect, 0 = predict mean

Cross-validation: never evaluate on training data. K-fold CV gives you a reliable estimate of generalization performance.

sklearn Pipeline: chains preprocessing → model into one object. Prevents data leakage (scaler fit only on training fold), makes deployment clean.

1import numpy as np
2from sklearn.datasets import make_classification
3from sklearn.model_selection import (train_test_split, cross_val_score,
4                                      StratifiedKFold, GridSearchCV)
5from sklearn.pipeline import Pipeline
6from sklearn.preprocessing import StandardScaler
7from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
8from sklearn.metrics import (classification_report, confusion_matrix,
9                              roc_auc_score, f1_score, make_scorer)
10import optuna
11
12X, y = make_classification(n_samples=2000, n_features=20,
13                            weights=[0.8, 0.2],   # imbalanced: 80/20
14                            random_state=42)
15X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
16                                                     stratify=y)
17
18# ── Full sklearn Pipeline ─────────────────────────────────────────
19pipeline = Pipeline([
20    ("scaler", StandardScaler()),           # step 1: scale
21    ("model",  RandomForestClassifier(      # step 2: model
22        n_estimators=200, random_state=42
23    )),
24])
25pipeline.fit(X_train, y_train)
26y_pred  = pipeline.predict(X_test)
27y_proba = pipeline.predict_proba(X_test)[:, 1]
28
29print(classification_report(y_test, y_pred))
30print(f"AUC-ROC: {roc_auc_score(y_test, y_proba):.4f}")
31
32# ── Stratified K-Fold (keeps class ratio in each fold) ────────────
33cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
34cv_auc = cross_val_score(pipeline, X, y, cv=cv,
35                         scoring="roc_auc", n_jobs=-1)
36print(f"CV AUC: {cv_auc.mean():.4f} ± {cv_auc.std():.4f}")
37
38# ── Optuna: modern hyperparameter tuning ──────────────────────────
39def objective(trial):
40    params = {
41        "model__n_estimators": trial.suggest_int("n_estimators", 50, 300),
42        "model__max_depth":    trial.suggest_int("max_depth", 3, 10),
43        "model__min_samples_leaf": trial.suggest_int("min_samples_leaf", 1, 10),
44    }
45    pipeline.set_params(**params)
46    return cross_val_score(pipeline, X_train, y_train,
47                           cv=3, scoring="roc_auc").mean()
48
49study = optuna.create_study(direction="maximize")
50study.optimize(objective, n_trials=30, show_progress_bar=True)
51print(f"\nBest AUC  : {study.best_value:.4f}")
52print(f"Best params: {study.best_params}")