Lasso

Lasso

1. Idea

Lasso uses $L₁$ penalty for automatic feature selection:

Diamond-shaped constraint pushes many $w_{i}$ exactly to 0.

2. Ridge vs Lasso

	Ridge ($L₂$)	Lasso ($L₁$)
Penalty	sum of $w_{i}²$	sum of $|w_{i}|$
Effect	shrinks	shrinks + zeroes
Solution	closed form	iterative (no closed form)
Use	correlated features	sparse selection

3. Geometry (one line)

$L₁$ ball has corners on axes — optimum hits corner → some weights 0. $L₂$ ball is smooth — optimum rarely on axis.

4. Practice

Standardize features first ($L₁$ is scale-sensitive). Pick $λ$ by CV. Elastic Net mixes both: best when $d$ huge + correlated.

5. Exam Points

• Lasso objective + sparsity — 5 marks, very frequent.
• Ridge vs Lasso table + diagram — 5 marks, very frequent.
• Why no closed form (need subgradient / coordinate descent) — 3 marks.