Building a classification tree

Decision trees are a separate class of models in machine learning. Although a decision tree alone can be considered a weak learner, combined with the power of ensemble learning such as bagging or boosting, decision trees get great performances. Before digging into ensemble learning models and how to regularize them, in this recipe, we will review how decision trees work and how to use them on a classification task on the iris dataset.

To give an intuition of the power of decision trees, let’s consider a use case. We would like to know whether to sell ice creams on the beach based on two input features: sun and temperature.

We have the data in Figure 4.1 and would like to train a model on it.

Figure 4.1 – A circle if we should sell ice creams as a function of sun and temperature and a cross if we shouldn’t

For a human, this seems quite easy. For a linear model though, not so much. If we try to use...