Ensemble learning

Ensemble learning is the practice of combining multiple predictors, or models, to create a more robust model. Models can either be of the same type (homogenous ensembles) or different types (heterogenous ensembles). Further, ensemble learning is not specific to decision trees and can be applied to any ML technique, including linear models, neural networks (NNs), and more.

The central idea behind ensemble learning is that by aggregating the results of many models, we compensate for the weaknesses of a single model.

Of course, training the same models on the same data is not helpful in an ensemble (as the models will have similar predictions). Therefore, we aim for diversity in the models. Diversity refers to the degree to which each model in the ensemble differs. A high-diversity ensemble has widely different models.

There are several ways we can ensure diversity in our ensemble. One method is to train models on different subsets of the training data. Each...