Writing a simple optimization routine

Performing optimization is a common data science task. In this recipe, we implement a simple optimization routine using the Marquardt algorithm. For more information on this, see http://mathworld.wolfram.com/Levenberg-MarquardtMethod.html or https://en.wikipedia.org/wiki/Levenberg%E2%80%93Marquardt_algorithm. In the process of optimizing, we will also discover how Julia handles linear algebra and numerical differentiation.

The basic idea of the procedure is the following. Given a twice differentiable function, 

, and some point, 

, we want to find another point, 

, such that

. By repeating this process, we want to reach the local the minimum of the 

 function.

The rule for finding

 is as follows:

The idea is that we mix two standard algorithms: the Newton algorithm, where the step is

, and the gradient descent algorithm, where the step is

, where 

 is a parameter. We accept 

 if it leads to a decrease in the value of

.

The additional rule is that if we have found...