Confidence Intervals & Hypothesis Testing

Hypothesis testing is a framework for making decisions from data.

Steps:
1. State H₀ (null hypothesis) and H₁ (alternative)
2. Choose significance level α (usually 0.05)
3. Compute test statistic
4. Compute p-value = P(seeing this result or more extreme | H₀ is true)
5. If p-value < α → reject H₀

p-value: NOT the probability H₀ is true. It's the probability of observing data this extreme IF H₀ were true.

z-test: use when n is large (≥30) OR population variance σ² is known.
z = (x̄ - μ₀) / (σ/√n)

t-test: use when n is small AND variance is unknown (most real cases).
t = (x̄ - μ₀) / (s/√n)  with ν = n-1 degrees of freedom

Confidence Interval: range of plausible values for the true parameter.
95% CI = x̄ ± t(0.025, n-1) × s/√n
"95% of CIs constructed this way will contain the true mean."

Type I error (α): reject H₀ when it's true (false positive)
Type II error (β): fail to reject H₀ when it's false (false negative)
Power = 1 - β: probability of correctly detecting an effect

1import numpy as np
2from scipy import stats
3
4np.random.seed(42)
5
6# ── One-sample t-test ─────────────────────────────────────────────
7# H₀: mean delivery time = 30 min
8# H₁: mean delivery time ≠ 30 min (two-tailed)
9deliveries   = np.random.normal(32, 8, 25)    # n=25, small sample
10null_mean    = 30
11
12t_stat, p_val = stats.ttest_1samp(deliveries, null_mean)
13print(f"One-sample t-test:")
14print(f"  Sample mean = {deliveries.mean():.2f}")
15print(f"  t-statistic = {t_stat:.4f}")
16print(f"  p-value     = {p_val:.4f}")
17print(f"  Decision    : {'Reject H₀' if p_val < 0.05 else 'Fail to reject H₀'}")
18
19# ── Manual t-test (understand what scipy does) ────────────────────
20n    = len(deliveries)
21xbar = deliveries.mean()
22s    = deliveries.std(ddof=1)
23se   = s / np.sqrt(n)
24t_manual = (xbar - null_mean) / se
25print(f"
26  Manual t = ({xbar:.2f} - {null_mean}) / {se:.4f} = {t_manual:.4f}")
27
28# ── Confidence Interval ───────────────────────────────────────────
29alpha    = 0.05
30t_crit   = stats.t.ppf(1 - alpha/2, df=n-1)
31ci_lower = xbar - t_crit * se
32ci_upper = xbar + t_crit * se
33print(f"
3495% Confidence Interval: ({ci_lower:.2f}, {ci_upper:.2f})")
35print(f"  Null value {null_mean} {'NOT in CI → reject H₀' if not (ci_lower <= null_mean <= ci_upper) else 'in CI → fail to reject'}")
36
37# ── Two-sample t-test (compare two groups) ────────────────────────
38# H₀: groups A and B have same mean
39group_A = np.random.normal(75, 10, 30)   # treatment
40group_B = np.random.normal(70, 12, 30)   # control
41
42t2, p2 = stats.ttest_ind(group_A, group_B)
43print(f"
44Two-sample t-test (A vs B):")
45print(f"  Mean A={group_A.mean():.2f}, Mean B={group_B.mean():.2f}")
46print(f"  t={t2:.4f}, p={p2:.4f}")
47print(f"  Decision: {'Reject H₀' if p2 < 0.05 else 'Fail to reject H₀'}")
48
49# ── z-test (large sample) ─────────────────────────────────────────
50large_sample = np.random.normal(32, 8, 200)
51z_stat = (large_sample.mean() - 30) / (8 / np.sqrt(200))
52p_z    = 2 * (1 - stats.norm.cdf(abs(z_stat)))
53print(f"
54z-test (n=200):")
55print(f"  z={z_stat:.4f}, p={p_z:.6f}")