Machine Learning
105602Module 4: Hypothesis Testing and Ensemble Methods
1(f). If the p-value is smaller than the significance level α, we generally (i) Reject H0 (ii) Accept H0 (iii) Increase α automatically (iv) Reject H120262m
Module 4: Hypothesis Testing and Ensemble Methods
View this question on its own page →If the p-value is smaller than the significance level α, we generally (i) Reject H0 (ii) Accept H0 (iii) Increase α automatically (iv) Reject H1
Worked SolutionAnswer: (i) Reject (H_0)
When the p-value is below the chosen significance level (\alpha), the observed data would be sufficiently unlikely under the null hypothesis, so we reject (H_0). This does not prove the alternative hypothesis with certainty.
1(g). AdaBoost primarily focuses subsequent learners on (i) Randomly selected features only (ii) Previously misclassified or difficult observations (iii) Only correctly classified observations (iv) Unlabelled observations20262m
Module 4: Hypothesis Testing and Ensemble Methods
View this question on its own page →AdaBoost primarily focuses subsequent learners on (i) Randomly selected features only (ii) Previously misclassified or difficult observations (iii) Only correctly classified observations (iv) Unlabelled observations
Worked SolutionAnswer: (ii) Previously misclassified or difficult observations
AdaBoost increases the weights of incorrectly classified training examples so subsequent weak learners pay more attention to them. Learners are combined using weights based on their performance.
7(a). Explain ensemble learning and its motivation. Why can combining multiple weak or unstable learners improve prediction performance?20267m
Module 4: Hypothesis Testing and Ensemble Methods
View this question on its own page →Explain ensemble learning and its motivation. Why can combining multiple weak or unstable learners improve prediction performance?
Worked SolutionEnsemble learning
Ensemble learning combines predictions from multiple models to produce a stronger predictor. It is useful because a single model may be unstable, biased, or sensitive to sample variation.
- Bagging: train models on bootstrap samples and average/vote; mainly reduces variance.
- Boosting: train learners sequentially, emphasizing previous errors; can reduce bias and build a strong learner.
- Voting/stacking: combine predictions or learn how to combine them.
Ensembles work best when component models are reasonably accurate and make different errors. If every learner makes identical errors, combining them provides little benefit. Diversity, proper validation and avoiding data leakage are important.
7(b). Explain the Random Forest algorithm. Discuss the role of bootstrap samples and random feature selection.20267m
Module 4: Hypothesis Testing and Ensemble Methods
View this question on its own page →Explain the Random Forest algorithm. Discuss the role of bootstrap samples and random feature selection.
Worked SolutionRandom Forest algorithm
Random Forest trains many decision trees and aggregates their outputs. For each tree, it draws a bootstrap sample (sample with replacement) from the training set. At each split, it considers only a random subset of features, then chooses the best split among them.
Classification uses majority vote; regression commonly averages tree predictions. Bootstrap sampling and random feature subsets decorrelate trees, reducing variance compared with a single deep tree. Out-of-bag examples can estimate generalization error without a separate validation set. Important limitations include reduced interpretability, memory cost and weaker performance on some highly structured extrapolation tasks.
9(d). Write a short note on Bagging and Boosting.20267m
Module 4: Hypothesis Testing and Ensemble Methods
View this question on its own page →Write a short note on Bagging and Boosting.
Worked SolutionBagging (bootstrap aggregating): train base learners independently on bootstrap samples and combine by voting or averaging. It mainly reduces variance; Random Forest is a bagging-based method with random feature selection.
Boosting: train learners sequentially so later learners focus on errors or residuals from earlier ones. AdaBoost reweights difficult observations; Gradient Boosting fits learners to loss gradients/residuals.
Bagging Boosting Learners can train in parallel Sequential dependency between rounds Bootstrap resampling Reweighting or residual fitting Primarily reduces variance Often reduces bias and builds strong predictors Less sensitive to individual noisy examples in many settings Can be sensitive to noise/outliers if overfit Both require validation and sensible tuning to avoid overfitting.
Q1(j). What is meant by Ensemble learning?20232m
Module 4: Hypothesis Testing and Ensemble Methods
View this question on its own page →What is meant by Ensemble learning?