LINEAR ALGEBRA / 3. MATRIX MULTIPLICATION

Matrix Multiplication

Composition of transformations — not just row × column

EXPLANATION

Matrix multiplication is the composition of two linear transformations. A @ B means "first apply B, then apply A."

Three ways to think about matrix multiplication:
1. Row × column: C[i,j] = Σₖ A[i,k] × B[k,j]  ← mechanical
2. Transformation composition: first B, then A  ← geometric (3B1B)
3. Column combinations: each column of AB is a linear combination of columns of A  ← algebraic

Rules:
• (m×k) @ (k×n) = (m×n)  — inner dimensions must match
• AB ≠ BA  — order matters! Transformations don't commute
• (AB)C = A(BC)  — associative
• A(B+C) = AB + AC  — distributive

In deep learning, the entire forward pass is just chained matrix multiplications:
Layer 1: Z₁ = X @ W₁ + b₁
Layer 2: Z₂ = A₁ @ W₂ + b₂
...
Output: ŷ = Aₙ₋₁ @ Wₙ + bₙ

The gradient of a matrix multiplication:
If Z = X @ W, then:
• dL/dW = Xᵀ @ dL/dZ
• dL/dX = dL/dZ @ Wᵀ

DIAGRAM

Matrix multiplication: A(m×k) @ B(k×n) = C(m×n)

  A = [[1,2],   B = [[5,6,7],
       [3,4]]        [8,9,10]]

  C[0,0] = row0(A) · col0(B) = 1*5 + 2*8 = 21
  C[0,1] = row0(A) · col1(B) = 1*6 + 2*9 = 24
  C[1,0] = row1(A) · col0(B) = 3*5 + 4*8 = 47

  Dimensions: (2×2) @ (2×3) = (2×3) ✓

  Batch matrix multiply (3D tensors):
  (batch, seq, d_model) @ (d_model, d_k) = (batch, seq, d_k)
  This is exactly the Q, K, V projections in attention!

CODE

PYTHON

1import numpy as np
2import torch
3
4# ── Basic matrix multiplication ───────────────────────────────────
5A = np.array([[1, 2], [3, 4]])
6B = np.array([[5, 6, 7], [8, 9, 10]])
7
8C = A @ B    # (2,2) @ (2,3) = (2,3)
9print(f"A shape: {A.shape}, B shape: {B.shape}")
10print(f"C = A @ B shape: {C.shape}")
11print(f"C =
12{C}")
13
14# Manual element: C[0,0] = row0 · col0
15print(f"
16C[0,0] = {A[0]} · {B[:,0]} = {A[0] @ B[:,0]}")
17
18# ── AB ≠ BA  (non-commutative) ────────────────────────────────────
19A = np.array([[1, 2], [0, 1]])
20B = np.array([[1, 0], [1, 1]])
21print(f"
22AB:
23{A @ B}")
24print(f"BA:
25{B @ A}")
26print(f"AB == BA: {np.allclose(A@B, B@A)}")   # False!
27
28# ── Gradients of matrix multiplication ───────────────────────────
29# If Z = X @ W, then: dL/dW = Xᵀ @ dL/dZ, dL/dX = dL/dZ @ Wᵀ
30np.random.seed(42)
31X = np.random.randn(3, 4)   # (batch=3, in_features=4)
32W = np.random.randn(4, 2)   # (in_features=4, out_features=2)
33Z = X @ W                   # (3, 2)
34dL_dZ = np.ones_like(Z)     # gradient from above (all ones for demo)
35
36dL_dW = X.T @ dL_dZ         # (4,3) @ (3,2) = (4,2)
37dL_dX = dL_dZ @ W.T         # (3,2) @ (2,4) = (3,4)
38print(f"
39Gradients of Z = X @ W:")
40print(f"  dL/dW shape: {dL_dW.shape} (same as W)")
41print(f"  dL/dX shape: {dL_dX.shape} (same as X)")
42
43# Verify with PyTorch autograd
44X_t = torch.tensor(X, requires_grad=True)
45W_t = torch.tensor(W, requires_grad=True)
46Z_t = X_t @ W_t
47Z_t.sum().backward()
48print(f"
49PyTorch verification:")
50print(f"  dL/dW matches: {np.allclose(X_t.grad.numpy(), dL_dX)}")  # dL/dX
51print(f"  dL/dX matches: {np.allclose(W_t.grad.numpy(), dL_dW)}")  # dL/dW
52
53# ── Batch matrix multiply (3D) ────────────────────────────────────
54# Used in transformer attention: (batch, seq, d_model) @ (d_model, d_k)
55batch, seq, d_model, d_k = 2, 5, 8, 4
56Q = np.random.randn(batch, seq, d_model)
57W_Q = np.random.randn(d_model, d_k)
58queries = Q @ W_Q      # (2, 5, 8) @ (8, 4) = (2, 5, 4)
59print(f"
60Batch matmul: {Q.shape} @ {W_Q.shape} = {queries.shape}")

← PREV2. Matrices & Transformations NEXT →4. Determinants