Conditional, Joint & Marginal Probability + Bayes Theorem

Conditional Probability: P(A|B) = P(A ∩ B) / P(B)
"The probability of A given that B has already occurred."

This restricts the sample space to B, then asks how much of that restricted space is A.

Joint Probability: P(A ∩ B) = P(A|B) × P(B) = P(B|A) × P(A)

Marginal Probability: P(A) = Σ P(A ∩ Bᵢ) for all mutually exclusive Bᵢ
"Sum over all ways A can happen" — law of total probability.

Bayes Theorem:
P(A|B) = P(B|A) × P(A) / P(B)

In ML terms:
P(hypothesis | data) = P(data | hypothesis) × P(hypothesis) / P(data)
    posterior        =    likelihood        ×    prior       /  evidence

This is the foundation of:
• Naive Bayes classifier
• Bayesian neural networks
• MCMC sampling
• Bayesian optimization (hyperparameter tuning)

1import numpy as np
2import matplotlib.pyplot as plt
3from fractions import Fraction
4
5# ── Bayes Theorem: medical test ───────────────────────────────────
6P_disease      = 0.01    # prior: 1% prevalence
7P_pos_given_D  = 0.95    # sensitivity
8P_pos_given_nD = 0.10    # false positive rate
9
10# Law of total probability
11P_positive = P_pos_given_D * P_disease + P_pos_given_nD * (1 - P_disease)
12
13# Bayes theorem
14P_disease_given_pos = (P_pos_given_D * P_disease) / P_positive
15
16print(f"P(Disease | Positive test) = {P_disease_given_pos:.4f}")
17print(f"Only {P_disease_given_pos*100:.1f}% — counterintuitive!")
18
19# ── How prior affects posterior ───────────────────────────────────
20prevalences  = np.linspace(0.001, 0.5, 200)
21posteriors   = []
22for prev in prevalences:
23    p_pos    = P_pos_given_D * prev + P_pos_given_nD * (1 - prev)
24    posterior = (P_pos_given_D * prev) / p_pos
25    posteriors.append(posterior)
26
27# Plot: prior vs posterior — shows how much prior matters
28plt.figure(figsize=(8, 4))
29plt.plot(prevalences, posteriors, color="#C084FC", linewidth=2)
30plt.axhline(0.5, color="gray", linestyle="--", alpha=0.5, label="50% threshold")
31plt.xlabel("Disease Prevalence (Prior)")
32plt.ylabel("P(Disease | Positive Test)")
33plt.title("How Prior Affects Posterior (Bayes)")
34plt.legend(); plt.tight_layout()
35plt.savefig("bayes_prior.png", dpi=150)
36
37# ── Naive Bayes classifier from scratch ──────────────────────────
38# Spam detection: P(spam|words) ∝ P(words|spam) × P(spam)
39class NaiveBayesSpam:
40    def fit(self, docs, labels):
41        self.classes   = list(set(labels))
42        self.log_prior = {}
43        self.log_like  = {}
44        n = len(labels)
45        for c in self.classes:
46            docs_c = [d for d, l in zip(docs, labels) if l == c]
47            self.log_prior[c] = np.log(len(docs_c) / n)
48            all_words = " ".join(docs_c).split()
49            word_count = {}
50            for w in all_words:
51                word_count[w] = word_count.get(w, 0) + 1
52            total = sum(word_count.values()) + len(word_count)
53            self.log_like[c] = {w: np.log((cnt+1)/total)
54                                for w, cnt in word_count.items()}
55
56    def predict(self, doc):
57        words = doc.split()
58        scores = {}
59        for c in self.classes:
60            scores[c] = self.log_prior[c]
61            for w in words:
62                scores[c] += self.log_like[c].get(w, -10)
63        return max(scores, key=scores.get)
64
65docs   = ["buy cheap pills now", "meeting at 3pm", "free money click here", "project update"]
66labels = ["spam", "ham", "spam", "ham"]
67nb = NaiveBayesSpam()
68nb.fit(docs, labels)
69print(nb.predict("free pills cheap"))    # spam
70print(nb.predict("update the project"))  # ham