Ensemble Methods — Why and How

Why do we use ensemble methods?

A single classification model may be unstable (small data changes → big model changes, e.g. decision trees), or may be too weak to express the true decision boundary. Ensembles combine several base models and aggregate their predictions to obtain a single, stronger and more stable classifier. Benefits:

1. Improved accuracy — averaging/voting corrects individual model errors; different models make different mistakes, so combining them reduces the overall error.
2. Reduced variance — averaging many unstable models (e.g., trees) smooths out their erratic behaviour.
3. Reduced bias — a committee of models can represent relationships that a single simple model cannot.
4. Robustness — less sensitive to noise, overfitting, and outliers; often better generalisation.

Key Requirement

For an ensemble to help, the base models must be accurate (better than random) and diverse (make different errors).

Bagging (Bootstrap Aggregating) — an ensemble method

Steps:

1. Generate $m$ bootstrap samples from the original training set, each obtained by sampling with replacement (some tuples repeated, others omitted).
2. Train a base classifier (e.g., a decision tree) on each sample — typically grown without pruning (high variance, but diverse).
3. Combine the $m$ classifiers by majority voting for classification (or averaging for regression).

Training set ──bootstrap──▶ S1 ──▶ C1 ─┐
                       S2 ──▶ C2 ─┼─▶ majority vote ─▶ final class
                       ...              │
                       Sm ──▶ Cm ─┘

Why it works: each bootstrap sample differs slightly, so the trees are diverse; voting reduces variance without increasing bias, so overall accuracy improves.

Other ensemble methods

• Random Forest = bagging + random attribute selection at each split.
• Boosting (AdaBoost) = sequential training that re-weights misclassified tuples and combines by weighted vote.
• Stacking = combine heterogeneous base learners with a meta-classifier.

Summary: we use ensembles to get higher, more stable accuracy than any single classifier; a classic simple ensemble is bagging, which trains diverse models on bootstrap samples and votes.