Optimization Techniques for Machine Learning

We discussed mathematical optimization techniques in the previous chapter and their necessity in business problems that require minimizing the cost (error) function and in predictive modeling, wherein the machine learns from historical data to predict the future. In Machine Learning (ML), the cost is a loss function or an energy function that is minimized. It can be challenging in most cases to know which optimization algorithm should be considered for a given ML model. Optimization is an iterative process to maximize or minimize an objective function and there is always a trade-off between the number of iteration steps taken and the computational hardship to get to the next step. In this chapter, hints of how to choose an optimization algorithm given a problem (hence, an objective) have been provided. The choice of optimization algorithm depends on different factors, including the specific problem to be solved, the size and complexity of...