Gradient Boosting

Gradient Boosting

1. Idea

Build additive model by fitting each new tree to residuals (gradients) of current ensemble — steepest descent in function space.

$η$ (learning rate, 0.05-0.1) shrinks steps; more trees compensate.

2. Algorithm (squared loss view)

F₀ = mean(y)
for m: r = y - F_{m-1} (residuals); fit shallow tree h_{m} on r; F_{m} = F_{m-1} + η h_{m}

General loss: $r$ = negative gradient of loss (e.g. $y - p$ for log-loss).

3. XGBoost / LightGBM Extras

Second-order (Newton) steps, $L₁/L₂$ on leaves, column subsampling, missing-value handling, early stopping. Tabular-data state of the art.

4. Tuning

Few deep trees overfit — use shallow (depth 3-6) + many trees + small $η$ + subsample 0.8. Monitor validation.

5. Exam Points

• Residual-fitting idea + update — 7 marks, very frequent.
• $η$, depth, subsample roles — 3-5 marks, frequent.
• Boosting vs forest (bias vs variance) — 3 marks, frequent.