After reading this chapter, you will be able to understand how several linear regression algorithms work and how to tune your linear regression model. You will also learn to use some parts of Math.NET and Accord.NET, which make implementing some of the linear regression algorithms simple. Along the way, you will also learn how to use FsPlot to plot various charts. All source code is made available at https://gist.github.com/sudipto80/3b99f6bbe9b21b76386d.

Based on the approach used and the number of input parameters, there are several types of linear regression algorithms to determine the real value of the target variable. In this chapter, you will learn how to implement the following algorithms using F#:

These algorithms will be implemented using a robust industry standard open source .NET mathematics API called Math.NET. Math.NET has an F# friendly wrapper.

In this chapter, you will learn how to use the preceding APIs to solve problems using several linear regression methods and plot the result.

FsPlot is a charting library for F# to generate charts using industry standard JavaScript charting APIs, such as HighCharts. FsPlot provides a nice interface to generate several combination charts, which is very useful when trying to understand the linear regression model. You can find more details about the API at its homepage at https://github.com/TahaHachana/FsPlot.

Math.NET Numerics for F# 3.7.0

Math.NET Numerics is the numerical foundation of the Math.NET project, aiming to provide methods and algorithms for numerical computations in science, in engineering, and in everyday use. It supports F# 3.0 on .Net 4.0, .Net 3.5, and Mono on Windows, Linux, and Mac; Silverlight 5 and Windows 8 with PCL portable profile 47; Android/iOS with Xamarin.

You can get the API from the NuGet page at https://www.nuget.org/packages/MathNet.Numerics.FSharp/. For...

Using Math.Net numerics, you can create matrices and vectors easily. The following section shows how. However, before you can create the vector or the matrix using Math.NET API, you have to reference the library properly. The examples in this chapter run using the F# script.

You have to write the following code at the beginning of the file and then run these in the F# interactive:

Creating a vector

You can create a vector as follows:

The vector values must always be float as per Math.NET. Once you run this in the F# interactive, it will create the following output:

The general linear regression model calculation requires us to find the inverse of the matrix, which can be computationally expensive for bigger matrices. A decomposition scheme, such as QR and SVD, helps in that regard.

QR decomposition breaks a given matrix into two different matrices—Q and R, such that when these two are multiplied, the original matrix is found.

In the preceding image, X is an n x p matrix with n rows and p columns, R is an upper diagonal matrix, and Q is an n x n matrix given by:

Here, Q1 is the first p columns of Q and Q2 is the last n – p columns of Q.

Using the Math.Net method QR you can find QR factorization:

Just to prove the fact that you will get the original matrix back, you can multiply Q and R to see if you get the original matrix back:

let myMatAgain = qr.Q * qr.R  

Let's say you have a list of data point pairs such as the following:

You want to find out if there are any linear relationships between and .

In the simplest possible model of linear regression, there exists a simple linear relationship between the independent variable (also known as the predictor variable) and the dependent variable (also known as the predicted or the target variable). The independent variable is most often represented by the symbol and the target variable is represented by the symbol . In the simplest form of linear regression, with only one predictor variable, the predicted value of Y is calculated by the following formula:

is the predicted variable for . Error for a single data point is represented by:

and are the regression parameters that can be calculated with the following formula.

The best linear model minimizes the sum of squared errors. This is known as Sum of Squared Error (SSE).

For the best model, the regression...

The following is an example problem that can be solved using linear regression.

For seven programs, the amount of disk I/O operations and processor times were measured and the results were captured in a list of tuples. Here is that list: (14,2), (16,5),(27,7) (42,9), (39, 10), (50,13), (83,20). The task for linear regression is to fit a model for these data points.

For this experiment, you will write the solution using F# from scratch, building each block one at a time.

The following is the final output we receive:

Now in order to understand how good your linear regression model fits the data, we need to plot the actual data points as scatter plots and the regression line as a straight...

In the previous example, we used F# to obtain b0 and b1 from the first principal. However, we can use Math.NET to find these values for us. The following code snippet does that for us in three lines of code:

This produces the following result in the F# interactive:

Take a moment to note that the values are exactly the same as we calculated earlier.

In this example, you will see how Math.NET and FsPlot can be used together to generate the linear regression coefficients and plot the result. For this example, we will use a known relation between Relative Humidity (RH) and Dew point temperature. The relationship between relative humidity and dew point temperature is given by the following two formulas:

Here, t and t_d are the temperatures in degrees Celsius.

t_d is dew point, which is a measure of atmospheric moisture. It is the temperature to which the air must be cooled in order to reach saturation (assuming the air pressure and the moisture content are constant).

Let's say the dew point is 10 degrees Celsius, then we can see how linear regression can be used to find a relationship between the temperature and RH.

The following code snippet generates a list of 50 random temperatures and then uses the formula to find the RH. It then feeds this data into a linear regression system to find the best...

Sometimes it makes more sense to include more predictors (that is, the independent variables) to find the value of the dependent variable (that is, the predicted variable). For example, predicting the price of a house based only on the total area is probably not a good idea. Maybe the price also depends on the number of bathrooms and the distance from several required facilities, such as schools, grocery stores, and so on.

So we might have a dataset as shown next, and the problem we pose for our linear regression model is to predict the price of a new house given all the other parameters:

In this case, the model can be represented as a linear combination of all the predictors, as follows:

Here, the theta values represent the parameters we must select to fit the model. In vectorized form, this can be written as:

Theta can be calculated by the following formula:

So using the MathNet.Fsharp package, this can be calculated as follows:

Previously, in Chapter 1, Introduction...

As mentioned earlier, a regression model can be found using matrix decomposition such as QR and SVD.

The following code finds the theta from QR decomposition for the same data:

let qrlTheta = MultipleRegression.QR( created2 ,Y_MPG)

When run, this shows the following result in the F# interactive:

Now using these theta values, the predicted miles per gallon for the unknown car can be found by the following code snippet:

let mpgPredicted = qrlTheta * vector[8.;360.;215.;4615.;14.]

Similarly, SVD can be used to find the linear regression coefficients as done for QR:

let svdTheta = MultipleRegression.SVD( created2 ,Y_MPG)

Sometimes each sample, or in other words, each row of the predictor variable matrix, is treated with different weightage. Normally, weights are given by a diagonal matrix where each element on the diagonal represent the weight for the row. If the weight is represented by W, then theta (or the linear regression coefficient) is given by the following formula:

Math.NET has a special class called WeightedRegression to find theta. If all the elements of the weight matrix are 1 then we have the same linear regression model as before.

The weight matrix is normally determined by taking a look at the new value for which the target variable has to be evaluated. If the new value is depicted as x, then the weights are normally calculated using the following formula:

The numerator of the weight matrix can be calculated as the Euclidean distance between two vectors. The first vector is from the training data and the other is the new vector depicting the new entry for which the...