Machine Learning Math Foundations (2) — Linear Algebra

Preface: linear algebra has many abstract topics; this note tries to connect them—if not in one chain, then two. Main objects: vectors, determinants, matrices, and systems of equations.

Viewpoint

The core problem is solving systems of multivariate equations. Core ideas: inner product, rank, matrix inverse. Applications: linear regression, least squares via QR decomposition, SVD, PCA via diagonalizable matrices.

Vectors

• Basics: a vector represents n independent properties (dimensions), often written with an arrow or as coordinates. Unit vector: magnitude 1. Inner (dot) product is a scalar: Orthogonal vectors: inner product zero.

• Applications: vector sets and eigenvectors

Matrices

Definition: parameters describing linear relationships—a matrix is a linear map from some vectors to others. Y = AX maps X to Y; A describes the map. Y = AX appears often in linear dependence of vector sets. Intuition:

• Matrices and vectors: when m = 1 or n = 1, A is a row or column vector.

• Square matrices: negative matrix, upper/lower triangular, diagonal, identity. Determinant transforms use triangular form. Distinguish from unit vectors.

• Matrix transpose

Determinants

A determinant is usually a scalar—a function measuring how a linear transform affects “area” or “volume.”

• Determinant of a square matrix: n×n matrix A has |A| or det(A).

• Cofactor: A_ij = (−1)^(i+j) M_ij Adjugate matrix—for finding the inverse Square matrix inverse: if AB = BA = E, B is the inverse of A; A is invertible (nonsingular). Otherwise A is singular. Inverse denoted A⁻¹: ## Elementary matrix operations Row and column elementary transforms. Row echelon form, reduced row echelon, standard form—the former for variable relations, the latter for rank. Theorem (1): if A becomes B by elementary row ops, there exists invertible P with … How to find P? Matrix rank: k-th order minor is a determinant ## Vector sets A vector set: a collection of finitely many row or column vectors of the same dimension. Linear representation of vectors: As a system: Similarly, if set B is representable by set A then: AX = B has a solution.

• Linear dependence and independence: Use rank to test dependence: ## Linear systems Theorem 1: homogeneous Ax = 0 has nontrivial solutions iff R(A) … Positive definite matrix: if xᵀAx > 0 for all x ≠ 0, A is positive definite.

• Orthogonal matrix: n×n A with AᵀA = E is orthogonal (unitary over ℂ). Equivalent: columns (rows) are unit vectors and pairwise orthogonal.

QR Decomposition (Orthogonal–Triangular)

For m×n column-full-rank A:

Uses Gram–Schmidt orthogonalization:

Singular Value Decomposition

Generalizes symmetric eigen-decomposition to arbitrary matrices.

Corresponding to eigenvalues/vectors: diagonal entries of Σ are singular values; U and V are left/right singular vector matrices.

• Equivalence canonical form:

• Steps: eigenvalues/vectors; eigenvectors form V₁, compute U₁:

Derivatives of Vectors

A is m×n, x is n×1 column vector; Ax is m×1.

• Partial derivative formulas:

• Scalar w.r.t. matrix: Postscript: Still learning—slow updates; we improve together.

You might also like: my NumPy notes for ML, my matplotlib notes for ML, my calculus notes for ML