ML as mathematical optimization

ML can be described as finding the unknown underlying (approximate) function that maps input examples to output examples. This is where the ML algorithm defines a parametrized mapping function and optimizes or minimizes the error in the function to find the values of its parameters. ML is function approximation along with function optimization. The function parameters are also called model coefficients. Each time we fit a model to a training dataset, we solve an optimization problem.

Each ML algorithm makes different assumptions about the form of the mapping function, which in turn influences the type of optimization to be performed. ML is a function approximation method to optimally fit input data. It is particularly challenging when the data (the size or the number of examples) is limited. An ML algorithm must be chosen in a way that it most efficiently solves an optimization problem; for example, SGD is used for neural nets, while ordinary least...