Ensemble of ensembles

Can we combine boosting and bagging? As we saw earlier, in bagging, we create multiple subsets of data and then train classifiers on those datasets. We can treat AdaBoost as a classifier while doing bagging. The process is simple: first, we create the bags and then train different AdaBoost classifiers on each bag. Here, AdaBoost is an ensemble in itself. Thus, these models are called an ensemble of ensembles.

On top of having an ensemble of ensembles, we can also do undersampling (or oversampling) at the time of bagging. This gives us the benefits of bagging, boosting, and random undersampling (or oversampling) in a single model. We will discuss one such algorithm in this section, called EasyEnsemble. Since random undersampling doesn’t have significant overhead, both algorithms have training times similar to any other algorithm with the same number of weak classifiers.

EasyEnsemble

The EasyEnsemble algorithm [8] generates balanced datasets from...