Bias and variance diagnosis

We aim to have a model with high performance, or low error, in the training set (that is, a low bias model) while keeping the performance high, or error low, for new data points (that is, a low variance model). As we don’t have access to unseen new data points, we must use validation and test sets to assess the variance or generalizability of our models. Model complexity is one of the important factors in determining the bias and variance of machine learning models. By increasing complexity, we let a model learn more complex patterns in training data that could reduce training errors or model bias (Figure 4.9):

Figure 4.9 – Error versus model complexity for (A) high bias, (B) high variance, and (C, D) two different cases of low bias and low variance models

This decrease in error helps build a better model, even for new data points. However, this trend changes after a point, and higher complexities could cause overfitting...