Descriptive Statistics

Descriptive statistics summarize and describe the key features of a dataset.

Measures of Central Tendency:
• Mean   → arithmetic average. Sensitive to outliers
• Median → middle value. Robust to outliers — use for skewed data
• Mode   → most frequent value. Used for categorical data

Measures of Spread:
• Range      → max - min. Sensitive to outliers
• Variance   → E[(X-μ)²] — average squared deviation from mean
• Std Dev    → √Variance — same units as data, more interpretable
• IQR        → Q3-Q1, robust spread measure

Sample vs Population:
• Population variance: σ² = Σ(xᵢ-μ)² / N
• Sample variance:     s² = Σ(xᵢ-x̄)² / (N-1)  ← Bessel's correction
  The (N-1) corrects for the bias from estimating μ with x̄

Skewness: measure of asymmetry
• Positive skew: tail on right, mean > median
• Negative skew: tail on left, mean < median

Kurtosis: measure of tail heaviness vs Normal distribution

1import numpy as np
2from scipy import stats
3import pandas as pd
4
5# ── Generate realistic dataset ────────────────────────────────────
6np.random.seed(42)
7salaries = np.concatenate([
8    np.random.normal(50000, 10000, 900),   # most employees
9    np.random.normal(200000, 30000, 100),  # executives
10])
11
12# ── Central tendency ──────────────────────────────────────────────
13mean   = np.mean(salaries)
14median = np.median(salaries)
15mode   = stats.mode(salaries.round(-3)).mode  # round to nearest 1000
16
17print("Central Tendency (skewed salary data):")
18print(f"  Mean   = ${mean:,.0f}")    # pulled up by executives
19print(f"  Median = ${median:,.0f}")  # more representative
20print(f"  Mode   ≈ ${mode:,.0f}")
21
22print(f"
23  Mean > Median → positive skew (as expected)")
24
25# ── Spread ────────────────────────────────────────────────────────
26print("
27Measures of Spread:")
28print(f"  Range    = ${salaries.max()-salaries.min():,.0f}")
29print(f"  Variance = {np.var(salaries, ddof=0):,.0f}  (population)")
30print(f"  Variance = {np.var(salaries, ddof=1):,.0f}  (sample, Bessel)")
31print(f"  Std Dev  = ${np.std(salaries, ddof=1):,.0f}")
32q1, q3 = np.percentile(salaries, [25, 75])
33print(f"  IQR      = ${q3-q1:,.0f}  (Q3-Q1)")
34
35# ── Skewness and kurtosis ─────────────────────────────────────────
36print(f"
37Shape:")
38print(f"  Skewness = {stats.skew(salaries):.4f}  (>0 = right tail)")
39print(f"  Kurtosis = {stats.kurtosis(salaries):.4f}  (excess kurtosis)")
40
41# ── Five number summary (pandas describe) ─────────────────────────
42s = pd.Series(salaries)
43print(f"
44Five-number summary:
45{s.describe().apply(lambda x: f'{x:,.0f}')}")
46
47# ── Effect of outliers ────────────────────────────────────────────
48clean  = np.random.normal(50000, 10000, 100)
49with_outlier = np.append(clean, 10_000_000)   # one billionaire
50
51print(f"
52Outlier effect:")
53print(f"  Mean without outlier: ${clean.mean():,.0f}")
54print(f"  Mean with outlier:    ${with_outlier.mean():,.0f}")  # explodes
55print(f"  Median without:       ${np.median(clean):,.0f}")
56print(f"  Median with:          ${np.median(with_outlier):,.0f}")  # stable