Ensemble Methods

Ensemble Methods

1. Idea

Combine many weak models into one strong model. Works when members are accurate (>50%) and diverse (different errors cancel by voting).

2. Why Variance Drops

Averaging $M$ uncorrelated models with variance $σ²$ gives variance $σ² / M$. Bias stays similar — ensembles fix variance, not bias (boosting fixes bias too).

3. Three Families

Method	How diversity arises	Example
Bagging	bootstrap samples + average	Random Forest
Boosting	sequential, focus on mistakes	AdaBoost, GBM
Stacking	different model types + meta-learner	RF + SVM → logistic

4. When To Use

Unstable learners (trees, nets) gain most. Stable (linear) gain little. Always CV-tune ensemble size.

5. Exam Points

• Define ensemble + diversity — 5 marks, very frequent.
• Variance reduction $σ²/M$ intuition — 3 marks, frequent.
• Bagging vs boosting vs stacking — 5 marks, frequent.