In this chapter, we presented one of the most powerful ensemble learning techniques, boosting. We presented two popular boosting algorithms, AdaBoost and gradient boosting. We presented custom implementations for both algorithms, as well as usage examples for the scikit-learn implementations. Furthermore, we briefly presented XGBoost, a library dedicated to regularized, distributed boosting. XGBoost was able to outperform all other methods and implementations on both regression as well as classification problems.
AdaBoost creates a number of base learners by employing weak learners (slightly better than random guessing). Each new base learner is trained on a weighted sample from the original train set. Weighted sampling from a dataset assigns a weight to each instance and then samples from the dataset, using the weights in order to calculate the probability that each instance...
