Ensembling decision trees – gradient-boosted trees

Boosting, which is another ensemble technique, takes an iterative approach instead of combining multiple learners in parallel. In boosted trees, individual trees are no longer trained separately. Specifically, in Gradient-Boosted Trees (GBT) (also called Gradient-Boosting Machines), individual trees are trained in succession where a tree aims to correct the errors made by the previous tree. The following two diagrams illustrate the difference between random forest and GBT.

The random forest model builds each tree independently using a different subset of the dataset, and then combines the results at the end by majority votes or averaging:

Figure 3.14: The random forest workflow

The GBT model builds one tree at a time and combines the results along the way:

Figure 3.15: The GBT workflow

GBT works by iteratively improving the ensemble’s predictions through the addition of sequentially trained...