Machine Learning (20) — Dimensionality Reduction

Preface: every outcome follows from causes; when building ML models, complex features often require dimensionality reduction. Below are the main principles.

Why Reduce Dimensions?

In real projects, feature selection/dimensionality reduction is often required because:

• Multicollinearity: correlated features destabilize the solution space and weaken generalization;

• High-dimensional spaces are sparse, making patterns hard to find;

• Too many variables obscure structure;

• Looking at single-variable effects may miss joint relationships.

Goals of feature selection/reduction:

• Reduce the number of features

• Keep features as independent as possible

Sometimes the feature matrix is simply too large—slow training, long runtime.

Common methods:

• PCA

• LDA

• Topic models for dimensionality reduction

PCA Principle

Principal Component Analysis (PCA): merge high-dimensional features into lower dimensions—unsupervised. Find a linear projection to a lower space maximizing variance (maximum variance theory), keeping much of the original structure with fewer dimensions.

• Choosing components: with n original dimensions, pick the direction of maximum variance as the first axis; the second orthogonal to the first; the third orthogonal to both; repeat until the desired dimension. These directions are “principal components.”

• Math: objective—maximize projected variance. Projection matrix W; point x_i projects to Wᵀx_i (assume X is centered, z-score). Spread projections across dimensions → maximize sum of variances: PCA objective: Lagrange solution: Partial derivatives: Treating XXᵀ as A, solving W is eigen-decomposition of A—PCA on centered data is eigenvalues/eigenvectors of the covariance matrix. Eigenvalues are often computed via SVD; XXᵀ is diagonal so eigen-decompose: SVD on X gives:

LDA

Linear Discriminant Analysis (LDA): supervised feature merging using class labels. Project labeled points to lower dimensions so classes form tight clusters—same class closer after projection. In one sentence: “minimize within-class variance, maximize between-class variance.”

*Math (minimize within-class variance, maximize between-class variance)

Let the projection be w; linear transform:

Projected data is one-dimensional.

For binary classification, threshold the projected value into classes. Class centers represent class info—maximize distance between centers:

Also require within-class samples to stay close—minimize within-class covariance of projections:

Combined objective:

Transform the objective (A, B square; A positive definite):

Same form as PCA’s optimization—eigen-decompose the middle matrix.

Comparison

Similarities:

• Both reduce dimensionality

• Both use matrix decomposition

• Both often assume Gaussian data

Differences:

• LDA is supervised; PCA is unsupervised

• LDA reduces to at most k−1 dimensions (k classes); PCA has no such cap

• LDA can also classify

• LDA picks the projection best for separation; PCA picks maximum variance direction