LINEAR ALGEBRA / 7. DOT PRODUCT & NORMS

Dot Product, Norms & Orthogonality

Measuring similarity, distance, and angles in vector space

EXPLANATION

The dot product is the most fundamental operation in ML. It appears in:
• Cosine similarity in RAG retrieval
• Attention scores in transformers: score = QKᵀ/√d_k
• Linear layer: z = Wx + b (each row of W dots with x)
• Perceptron: fires if w·x + b > 0

Dot product formula:
a · b = Σ aᵢbᵢ = ||a|| · ||b|| · cos(θ)

Orthogonality: a·b = 0 → vectors are perpendicular.
Orthonormal: vectors are both orthogonal AND unit length.

Norms measure vector size:
• L1: Σ|xᵢ| → "Manhattan distance," leads to sparse solutions (Lasso)
• L2: √(Σxᵢ²) → "Euclidean length," standard distance
• L∞: max|xᵢ| → worst-case size

Orthogonal matrices Q (where QQᵀ = I):
• Columns are orthonormal vectors
• ||Qx|| = ||x|| → preserves vector lengths
• det(Q) = ±1 → pure rotation (or reflection)
• Qᵀ = Q⁻¹ → easy to invert!

Gram-Schmidt: algorithm to orthogonalize any set of vectors.

DIAGRAM

Dot product geometry:
  a·b = ||a||·||b||·cos(θ)

  Parallel (θ=0):    a·b = ||a||·||b||  (max)
  Perpendicular(θ=90°): a·b = 0
  Opposite (θ=180°): a·b = -||a||·||b|| (min)

  Cosine similarity in RAG:
  query_vec  = embed("how does RAG work?")
  chunk_vec1 = embed("RAG retrieves relevant context...")
  chunk_vec2 = embed("Paris is the capital of France")

  sim1 = cos_sim(query, chunk1) ≈ 0.89  (relevant!)
  sim2 = cos_sim(query, chunk2) ≈ 0.12  (irrelevant)

  Attention: scores = Q @ Kᵀ / √d_k
  Each row of scores = one query dotted with all keys

CODE

PYTHON
1import numpy as np
2
3# ── Dot product properties ────────────────────────────────────────
4a = np.array([1., 2., 3.])
5b = np.array([4., 0., -1.])
6
7dot_ab     = np.dot(a, b)
8cos_theta  = dot_ab / (np.linalg.norm(a) * np.linalg.norm(b))
9angle_deg  = np.degrees(np.arccos(np.clip(cos_theta, -1, 1)))
10
11print(f"a · b = {dot_ab}")
12print(f"cos(θ) = {cos_theta:.4f}")
13print(f"θ = {angle_deg:.2f}°")
14
15# ── Transformer attention mechanism ───────────────────────────────
16d_k  = 4
17Q    = np.random.randn(3, d_k)   # 3 queries
18K    = np.random.randn(5, d_k)   # 5 keys
19V    = np.random.randn(5, d_k)   # 5 values
20
21# Attention scores: how much each query "attends to" each key
22scores  = Q @ K.T / np.sqrt(d_k)  # (3, 5)
23weights = np.exp(scores) / np.exp(scores).sum(axis=1, keepdims=True)  # softmax
24output  = weights @ V              # (3, 4) — weighted sum of values
25
26print(f"
27Attention:")
28print(f"  Q: {Q.shape}, K: {K.shape}, V: {V.shape}")
29print(f"  Scores: {scores.shape}, Output: {output.shape}")
30
31# ── Gram-Schmidt orthogonalization ────────────────────────────────
32def gram_schmidt(vectors):
33    """Orthonormalize a set of vectors."""
34    orthonormal = []
35    for v in vectors:
36        # Subtract projections onto existing orthonormal vectors
37        for u in orthonormal:
38            v = v - np.dot(v, u) * u
39        # Normalize
40        norm = np.linalg.norm(v)
41        if norm > 1e-10:
42            orthonormal.append(v / norm)
43    return np.array(orthonormal)
44
45vecs  = [np.array([1., 1., 0.]),
46         np.array([1., 0., 1.]),
47         np.array([0., 1., 1.])]
48
49Q = gram_schmidt(vecs)
50print("
51Gram-Schmidt orthogonalization:")
52print(f"  Q =
53{Q.round(4)}")
54print(f"  QQᵀ (should be I):
55{(Q @ Q.T).round(4)}")
56
57# ── Norms and regularization ──────────────────────────────────────
58w = np.array([3., -1., 0., 2., 0., -4.])
59print(f"
60Weight vector: {w}")
61print(f"  L1 norm ||w||₁ = {np.sum(np.abs(w)):.2f}  (Lasso: promotes sparsity)")
62print(f"  L2 norm ||w||₂ = {np.linalg.norm(w):.4f}  (Ridge: shrinks all weights)")
63print(f"  L∞ norm ||w||∞ = {np.max(np.abs(w)):.2f}  (max element)")
← PREV6. Eigenvalues & EigenvectorsNEXT →8. SVD & Decompositions