Summary

We covered a lot of important topics in this chapter, starting with the foundation for tree-based ML models: the decision tree. We saw how trees can automatically determine splits for the data in order to make the best possible predictions, which uses a calculation such as the Gini coefficient or entropy. Next, we saw how these trees can be combined into an ensemble to form a random forest. Remember also that random forests bootstrap data for each decision tree, adding another element to prevent overfitting. Next, we saw how decision trees are used in ML boosting algorithms, such as AdaBoost, gradient boosting, XGBoost, LightGBM, and CatBoost.

These boosted algorithms fit decision trees to the data one step at a time, and each new tree fits to the data with weight added to incorrect predictions (AdaBoost) or fits to the gradient of the loss function (gradient boosting methods) in order to improve the model. We saw how the pycaret package can use the most prominent boosting...