Optimization

Optimization

1. Role

Training = minimize loss $J(w)$. All deep / linear solvers are optimization loops: pick direction, step, repeat.

2. Gradient Descent Family

• $η$ = learning rate. Too big → diverge; too small → slow.
• Batch: full data per step (exact, slow).
• Stochastic (SGD): one example per step (noisy, fast, escapes shallow minima).
• Mini-batch: 32-256 examples (standard).

init w; repeat: g = grad on batch; w = w - η g; until converged

3. Improvements

• Momentum: average past gradients (damps zigzag).
• Adaptive ($Adam$): per-weight $η$ from gradient history — default for nets.
• Standardize features (zero mean, unit variance) or conditioning is poor.

4. Closed Form vs Iterative

Small linear problems: solve directly. Large / non-convex: iterate. Always monitor train + validation loss curves.

5. Exam Points

• Write GD update + explain $η$ — 5 marks, very frequent.
• Batch vs SGD vs mini-batch — 3-5 marks, frequent.
• Why standardize + Adam helps — 3 marks, frequent.