Bagging and Boosting (Ensemble Methods):

Both combine weak learners (e.g., decision trees) to strong ensemble but differ in strategy.

Bagging (Bootstrap Aggregating) — Breiman:

• Create $M$ bootstrap samples (sample with replacement, size $n$) from training data.
• Train one base learner on each sample in parallel, e.g., deep trees in Random Forest.
• Aggregate: Classification majority vote, Regression averaging: $\hat f(x)=\frac1M\sum f_m(x)$
• Reduces variance, prevents overfitting, robust to noise.
• Example: Random Forest bagging 100 trees on loan default data, each tree sees 63% unique rows + random feature subset, vote gives stable prediction. Out-of-bag error estimates generalization without CV.

Boosting (e.g., AdaBoost, Gradient Boosting):

• Sequential. Train weak learners (stumps) iteratively; each new learner focuses on previously misclassified weighted examples.
• AdaBoost: Maintain weights $w_i$; initially $1/n$, increase weight for misclassified, decrease for correct. Final prediction weighted vote $F(x)=sign(\sum \alpha_m f_m(x))$ where $\alpha_m=\frac12\ln((1-e_m)/e_m)$.
• Gradient Boosting: Fits new learner to residual (negative gradient of loss) $r_i = -\partial L(y_i, F_{m-1}(x_i))/\partial F_{m-1}$.
• Reduces bias (+ variance), converts weak → strong, but prone to overfit if noisy; sequential not easily parallel.

Comparison:

	Bagging	Boosting
Data	Bootstrap subsets	Weighted full data, adaptive
Training	Parallel, independent	Sequential, dependent
Reduces	Variance	Bias
Sensitivity to noise	Robust	Sensitive
Example learners	Full trees	Shallow stumps

Example: Bagging on noisy medical data stable; Boosting on same achieves higher accuracy but may overfit unless early stopping/shrinkage.