Ridge Regression

Ridge Regression

1. Problem It Fixes

OLS explodes when features correlate or $d > n$ ($X^{T}X$ singular). Ridge adds $L₂$ penalty:

$λ ≥ 0$ controls shrinkage. Larger $λ$ → smaller, stabler $w$.

2. Closed Form

$λI$ makes matrix invertible even when $X^{T}X$ is singular. Bayesian view: Gaussian prior on $w$ (MAP).

3. Effect

$λ$	Behaviour
$0$	OLS (may overfit)
small	stabilizes, tiny bias
large	heavy shrinkage, may underfit

Ridge shrinks but never zeroes — keeps all features. Use cross-validation to pick $λ$.

4. Worked Intuition

two almost-identical features → OLS gives +1000, -999 (cancels)
Ridge λ=1 → gives +2.1, +1.9 (stable, small)

5. Exam Points

• Write objective + solution — 5-7 marks, very frequent.
• Why $λI$ fixes singularity — 3 marks, frequent.
• Ridge vs OLS, role of $λ$ — 3 marks, frequent.