Loss Functions and Optimization

In Chapter 2 and Chapter 3, we focused on the two most important and core math concepts that are at the heart of virtually all of data science. In this chapter, we are going to move on to math concepts behind specific, but still very important, data science activities. Specifically, we are going to lay some of the groundwork for building predictive models.

At the end of the last chapter, we hinted that one of the key concepts when building models is knowing or measuring how good a model is. When we train or fit a machine learning (ML) model, we adjust the parameter values of the model so that it gives a “better” fit or explanation of the data. But this raises the question: What do we mean by “better”? Without an exact quantitative definition of what we mean when we say that one set of parameter values gives a better fit to the data than another, we cannot construct an objective and quantitative training process. This is...