How gradient boosting works
In this section, we will look under the hood of gradient boosting and build a gradient boosting model from scratch by training new trees on the errors of the previous trees. The key mathematical idea here is the residual. Next, we will obtain the same results using scikit-learn's gradient boosting algorithm.
Residuals
The residuals are the difference between the errors and the predictions of a given model. In statistics, residuals are commonly analyzed to determine how good a given linear regression model fits the data.
Consider the following examples:
Bike rentals
a) Prediction: 759
b) Result: 799
c) Residual: 799 - 759 = 40
Income
a) Prediction: 100,000
b) Result: 88,000
c) Residual: 88,000 –100,000 = -12,000
As you can see, residuals tell you how far the model's predictions are from reality, and they may be positive or negative.
Here is a visual example displaying the residuals of a linear regression line:
...
