A brief introduction to ensemble algorithms

The idea that ensembles of models can outperform single ones is not a recent one. We can trace it back to Sir Francis Galton, who was alive in Victorian Britain. He figured out that, in order to guess the weight of an ox at a county fair, it was more useful to take an average from a host of more or less educated estimates from a crowd than having a single carefully devised estimate from an expert.

In 1996, Leo Breiman formalized the idea of using multiple models combined into a more predictive one by illustrating the bagging technique (also called the “bootstrap aggregating” procedure) that later led to the development of the even more effective random forests algorithms. In the period that followed, other ensembling techniques such as gradient boosting and stacking were also presented, thus completing the range of ensemble methods that we use today.

You can refer to a few articles to figure out how these ensembling...