Central Limit Theorem & Sampling

Central Limit Theorem (CLT): The most important theorem in statistics.

"If you take sufficiently large random samples from ANY population (with finite mean μ and variance σ²), the distribution of sample means will be approximately Normal, regardless of the original distribution."

Formally: X̄ ~ N(μ, σ²/n) as n → ∞

Where:
• X̄ = sample mean
• μ = population mean
• σ²/n = variance of the sampling distribution
• σ/√n = standard error (SE)

This is why:
• We can use z-tests and t-tests on non-normal data
• Averages of measurements are normally distributed
• Neural network gradients are approximately normal (sum of many small contributions)

Key insight: it's the SAMPLE MEAN that becomes normal, not the original data.

Standard Error = σ/√n → as sample size increases, sample mean becomes more precise.

Law of Large Numbers: as n→∞, sample mean X̄ → population mean μ.

1import numpy as np
2from scipy import stats
3import matplotlib.pyplot as plt
4
5np.random.seed(42)
6
7# ── Demonstrate CLT with exponential population ───────────────────
8population_lambda = 0.5
9population        = np.random.exponential(1/population_lambda, size=100_000)
10
11print(f"Original Exponential distribution:")
12print(f"  Mean  = {population.mean():.4f} (true = {1/population_lambda})")
13print(f"  Skew  = {stats.skew(population):.4f}  (highly skewed)")
14
15# Sample means for different n
16fig, axes = plt.subplots(2, 3, figsize=(14, 8))
17sample_sizes = [1, 2, 5, 10, 30, 100]
18
19for ax, n in zip(axes.flat, sample_sizes):
20    # Draw 10,000 samples of size n, compute mean of each
21    sample_means = [population[np.random.choice(len(population), n)].mean()
22                    for _ in range(10_000)]
23    sample_means = np.array(sample_means)
24
25    ax.hist(sample_means, bins=50, density=True, color="#7C3AED", alpha=0.7)
26
27    # Overlay theoretical normal
28    mu_theory = 1/population_lambda
29    se_theory = (1/population_lambda) / np.sqrt(n)
30    x = np.linspace(sample_means.min(), sample_means.max(), 200)
31    ax.plot(x, stats.norm.pdf(x, mu_theory, se_theory),
32            "r-", linewidth=2, label="Normal fit")
33    ax.set_title(f"n={n}, SE={se_theory:.3f}")
34    ax.set_xlabel("Sample Mean")
35    ax.legend(fontsize=7)
36
37plt.suptitle("Central Limit Theorem — Exponential Population", fontsize=13)
38plt.tight_layout()
39plt.savefig("clt_demo.png", dpi=150)
40
41# ── Standard Error and sample size ───────────────────────────────
42print("
43Standard Error vs Sample Size:")
44sigma = 1/population_lambda   # population std dev
45for n in [1, 4, 9, 16, 25, 100]:
46    se = sigma / np.sqrt(n)
47    print(f"  n={n:4d} → SE = {se:.4f}  (precision {sigma/se:.1f}× better than n=1)")
48
49# ── Law of Large Numbers ──────────────────────────────────────────
50running_means = np.cumsum(population[:5000]) / np.arange(1, 5001)
51print(f"
52Law of Large Numbers:")
53print(f"  After n=10:   mean={running_means[9]:.4f}")
54print(f"  After n=100:  mean={running_means[99]:.4f}")
55print(f"  After n=1000: mean={running_means[999]:.4f}")
56print(f"  True mean:    {1/population_lambda:.4f}")