Partial Least Squares

Partial Least Squares

1. Difference from PCA

PCA maximizes variance of $X$ alone (ignores $y$). PLS maximizes covariance between $X$ and $y$ — components predict $y$ better, especially when $d > n$ (chemometrics, genomics).

2. Idea in One Pass

Find direction $w$ blending high-variance + high-correlation with $y$, project out, repeat for next component. First PLS direction ∝ correlation-weighted features.

w ∝ X^{T} y (first component weights = feature-label covariance)
t = X w (scores) → deflate X, y → repeat

3. PLS vs PCA vs Ridge

Method	Uses $y$?	Handles $d>n$?
PCA + regression	No	Yes (but may keep useless variance)
PLS	Yes	Yes (fewer components)
Ridge	Yes	Yes (no reduction, keeps all)

4. Practice

Pick component count by CV. Standardize $X$. Stop when validation error rises.

5. Exam Points

• PLS vs PCA (covariance vs variance) — 5 marks, very frequent.
• When PLS wins ($d>n$, correlated $X$) — 3 marks, frequent.
• Component selection by CV — 2 marks.