Summary
In this chapter, we gained knowledge about which optimization algorithm must be considered to minimize (continuous) objective functions that are generally encountered in ML models. Such models have a real-valued evaluation of the input variables and involve local search. The differentiability of an objective function is perhaps the most important factor when considering the optimization algorithm type for a given problem.
The chapter did not contain an exhaustive list of algorithms to optimize ML models but captured the essence of the main ones and their underlying behavior with examples. It also touched upon the concepts of deterministic optimization and stochastic optimization, the latter encompassing GAs, whose utility is evolving in real-world problems.
