Principal Component Analysis

Principal Component Analysis

1. Goal

Compress $R^{d} → R^{k}$ ($k < d$) keeping maximum variance. Denoises, visualizes, speeds later models.

2. Steps

1. center X (subtract mean)
2. covariance C = (1/n) X^{T} X
3. eigen-decompose C → eigenvalues λ₁ ≥ λ₂ ...
4. keep top-k eigenvectors V_{k}
5. project Z = X V_{k}

Variance kept by top-$k$:

3. Worked

2-D elongated cloud: $λ₁$ large (long axis), $λ₂$ small. Keep $k=1$ → points on line, 95% variance kept. Choose $k$ for 90-99% variance.

4. Properties

• Components uncorrelated, ordered by variance.
• Sensitive to scale → standardize first.
• Linear only (kernel PCA for curves).

5. Exam Points

• PCA steps + variance ratio — 7 marks, very frequent.
• Eigen of covariance role — 3-5 marks, frequent.
• Why center / standardize — 3 marks, frequent.