Correlation & Covariance

Covariance measures how two variables change together.
Cov(X,Y) = E[(X-μₓ)(Y-μᵧ)] = E[XY] - E[X]E[Y]

• Cov > 0 → X and Y tend to increase together
• Cov < 0 → X increases when Y decreases
• Cov = 0 → no linear relationship

Problem: covariance depends on units and scale. Hard to interpret magnitude.

Pearson Correlation: ρ = Cov(X,Y) / (σₓ × σᵧ)
• Normalized version of covariance
• Always between -1 and +1
• +1 = perfect positive linear relationship
• -1 = perfect negative linear relationship
•  0 = no linear relationship

Spearman Correlation: rank-based version. Works for monotonic (not just linear) relationships. Robust to outliers. Use when data isn't normally distributed.

CRITICAL: Correlation ≠ Causation. Ice cream sales and drowning both correlate with summer — confounding variable (temperature).

In ML: correlated features cause multicollinearity, inflating coefficients in linear models. Remove or combine correlated features.

1import numpy as np
2from scipy import stats
3import matplotlib.pyplot as plt
4import seaborn as sns
5import pandas as pd
6
7np.random.seed(42)
8
9# ── Generate correlated data ──────────────────────────────────────
10n = 200
11x = np.random.normal(0, 1, n)
12y_pos  = 2*x + np.random.normal(0, 0.5, n)   # strong positive
13y_neg  = -x  + np.random.normal(0, 0.8, n)   # negative
14y_none = np.random.normal(0, 1, n)            # no correlation
15
16# ── Covariance ────────────────────────────────────────────────────
17cov_pos  = np.cov(x, y_pos)[0,1]
18cov_neg  = np.cov(x, y_neg)[0,1]
19cov_none = np.cov(x, y_none)[0,1]
20print(f"Cov(x, y_pos)  = {cov_pos:.4f}")
21print(f"Cov(x, y_neg)  = {cov_neg:.4f}")
22print(f"Cov(x, y_none) = {cov_none:.4f}")
23
24# ── Pearson correlation ───────────────────────────────────────────
25r_pos,  p_pos  = stats.pearsonr(x, y_pos)
26r_neg,  p_neg  = stats.pearsonr(x, y_neg)
27r_none, p_none = stats.pearsonr(x, y_none)
28print(f"
29Pearson correlations:")
30print(f"  r(x, y_pos)  = {r_pos:.4f}  (p={p_pos:.4f})")
31print(f"  r(x, y_neg)  = {r_neg:.4f}  (p={p_neg:.4f})")
32print(f"  r(x, y_none) = {r_none:.4f}  (p={p_none:.4f})")
33
34# ── Spearman correlation (rank-based, robust) ─────────────────────
35rs, ps = stats.spearmanr(x, y_pos)
36print(f"
37Spearman r(x, y_pos) = {rs:.4f}")
38
39# ── Correlation matrix on real-ish data ──────────────────────────
40df = pd.DataFrame({
41    "study_hours": np.random.normal(5, 2, 100),
42    "sleep_hours": np.random.normal(7, 1, 100),
43    "score":       None,
44    "stress":      None,
45})
46df["score"] = 60 + 5*df["study_hours"] - 2*df["stress"].fillna(0) + np.random.normal(0, 5, 100)
47df["stress"] = 8 - df["sleep_hours"] + np.random.normal(0, 0.5, 100)
48df["score"]  = 60 + 5*df["study_hours"] - 2*df["stress"] + np.random.normal(0, 5, 100)
49
50corr_matrix = df.corr()
51print(f"
52Correlation matrix:
53{corr_matrix.round(3)}")
54
55# Heatmap
56plt.figure(figsize=(6,5))
57sns.heatmap(corr_matrix, annot=True, cmap="RdPu", center=0,
58            fmt=".2f", square=True)
59plt.title("Correlation Matrix")
60plt.tight_layout()
61plt.savefig("correlation_heatmap.png", dpi=150)