LINEAR ALGEBRA / OVERVIEW

Linear Algebra — The Full Map

The language of data — vectors, matrices, and transformations

EXPLANATION

Linear algebra is the mathematics of vectors and linear transformations. For machine learning, it is the primary language — data is vectors, models are matrices, training is transformations.

The 3Blue1Brown mental model (Essence of Linear Algebra series):
• Vectors are arrows in space — they have direction and magnitude
• Matrices are transformations — they rotate, scale, shear space
• Matrix multiplication = applying one transformation after another
• Determinant = how much a transformation scales area/volume
• Eigenvectors = the special vectors that only get scaled, not rotated
• Dot product = how much two vectors point in the same direction

Why linear algebra for ML:
• Data → n-dimensional vectors (each feature = one dimension)
• Weights → matrices that transform data
• Neural network forward pass → sequence of matrix multiplications
• PCA → eigendecomposition of covariance matrix
• SVD → the most useful matrix decomposition in all of data science
• Attention mechanism → dot products between query/key vectors

DIAGRAM

VECTORS & SPACES         MATRIX OPERATIONS
  ─────────────────────    ──────────────────────────
  Vector notation          Matrix multiplication
  Vector addition          Transpose
  Scalar multiplication    Inverse
  Dot product              Determinant
  Norms (L1, L2)           Rank
  ─────────────────────    ──────────────────────────
  DECOMPOSITIONS           ML CONNECTION
  Eigendecomposition       Data = matrix of vectors
  SVD                      Weights = transform matrix
  PCA                      Attention = dot products
  Cholesky                 Embeddings = vector space
  ─────────────────────    Gradients = vectors in weight space

CODE

PYTHON

1import numpy as np
2
3# The core of linear algebra in one example:
4# A neural network forward pass IS matrix multiplication
5
6# Input: batch of 3 samples, each with 4 features
7X = np.array([[1, 2, 3, 4],
8              [5, 6, 7, 8],
9              [9, 10, 11, 12]])   # (3, 4)
10
11# Weight matrix: transform from 4 features to 2 hidden units
12W = np.random.randn(4, 2)        # (4, 2)
13b = np.zeros(2)                  # bias
14
15# Forward pass = matrix multiplication
16Z = X @ W + b    # (3, 4) @ (4, 2) = (3, 2)
17
18print(f"Input shape:  {X.shape}")
19print(f"Weight shape: {W.shape}")
20print(f"Output shape: {Z.shape}")
21print("Every neural network layer is just X @ W + b")

NEXT →1. Vectors & Operations