Bagging vs AdaBoost vs Gradient Boosting:

Bagging (Bootstrap Aggregating):

• Parallel, bootstrap samples, full trees, reduces variance by averaging (Random Forest). Each model independent, weight equal. Robust to overfit, handles noisy data well. Example: $M=100$ trees vote.

AdaBoost (Adaptive Boosting — Freund & Schapire):

• Sequential, adaptively reweights samples: increase weight for misclassified, decrease for correct. Weak learners typically stumps ($depth=1$). Final: $F(x)=\sum \alpha_t h_t(x)$, $\alpha_t=\frac12\ln((1-\epsilon_t)/\epsilon_t)$. Minimizes exponential loss $\sum \exp(-yF(x))$.
• Reduces bias (and some variance), very effective but sensitive to noise/outliers (weights blow up).

Gradient Boosting (Friedman — GBM, XGBoost):

• Sequential, fits new learner to residual / negative gradient of loss: $r_i^{(m)} = -\partial L(y_i,F_{m-1}(x_i))/\partial F_{m-1}$. For squared loss $r=y-F$, for logistic gradient. Update $F_m=F_{m-1}+
  u \gamma_m h_m(x)$($
  u$ shrinkage).
• Reduces bias, flexible (any differentiable loss), often strongest accuracy, needs careful tuning ( $M$, learning rate, depth, regularization) to avoid overfit. XGBoost adds second-order Taylor, leaf weight regularization $\Omega=\gamma T+\frac12\lambda||w||^2$.

Bias-Variance:

• Bagging — primarily variance reduction (averaging decorrelated trees); slight bias increase if bootstrap.
• Boosting — primarily bias reduction (sequentially correcting errors); variance can increase if too many iterations, controlled by shrinkage/early stopping.

Choosing: Noisy, limited tuning time → Bagging/RF. Max accuracy, structured data → Gradient Boosting/XGBoost/LightGBM. Very weak stumps + exponential loss → AdaBoost (now largely superseded by GBM).