Building regression trees

Before digging into the regularization of decision trees in general, let’s have a recipe for regression trees. Indeed, all the explanations in the previous recipe were assuming we have a classification task. Let’s explain how to apply it to a regression task and apply it to the California housing dataset.

For regression trees, only a few steps need to be modified compared to classification trees: the inference and the loss computation. Besides that, the overall principle is the same.

The inference

In order to make an inference, we can no longer use the most represented class in a leaf (or in the case of pure leaf, the only class). So, we use the average of the labels in each node.

In the example proposed in Figure 4.11, assuming this is a leaf, we would have an inference value that is the average of those 10 values equal to 14 in this case.

Figure 4.11 – The example of 10 samples with associated values...