Summary

In this chapter, we learned about a foundational optimization algorithm and its variants used in training ML and DL models. An application of the optimization technique in Python to a linear regression problem was also elaborated on. Both the cost function and its gradient, and how to update the gradient to converge to the optimal point, are mathematical concepts every data scientist must understand thoroughly; optimizing a cost function is the basis of achieving an optimal model for a problem or predictions. Different ways can be used to estimate the gradients depending on the behavior of the cost function.

In the following chapter, we will explore another fundamental algorithm, known as support vector machines (SVMs). Although SVMs can be used for regression problems, they are more widely used for classification tasks.