Common numerical optimization algorithms
Several numerical optimization algorithms are implemented in open source and commercially sold optimization software tools. A lot of these algorithms are based on heuristic search, which is a technique based on solving problems quickly compared to classic methods. Heuristics-based algorithms attempt to find an approximate solution since the exact solution is very hard to find. The solutions provided by heuristics-based methods are considered good enough to solve the problem; however, it is generally not the best solution. In this section, we will briefly discuss a few of these algorithms. For detailed discussions on these algorithms and their mathematical formulation, you can refer to the articles and texts cited in the Further reading section of this chapter.
Random restart hill climbing
In hill climbing, we start from a point, 
, and search in the neighborhood of 
. If the value of the objective function, 
, increases in any direction in...
