In this article by Tomasz Drabas, author of the book, Practical Data Analysis Cookbook, we will discuss on how Support Vector Machine models can be used as a classification engine.

(For more resources related to this topic, see here.)

Support Vector Machines

Support Vector Machines (SVMs) are a family of extremely powerful models that can be used in classification and regression problems. They aim at finding decision boundaries that separate observations with differing class memberships.

While many classifiers exist that can classify linearly separable data (for example, logistic regression), SVMs can handle highly non-linear problems using a kernel trick that implicitly maps the input vectors to higher-dimensional feature spaces. The transformation rearranges the dataset in such a way that it is then linearly solvable.

The mechanics of the machine

Given a set of n points of a form (x₁,y₁)...(x_n,y_n), where x_i is a z-dimensional input vector and  y_i is a class label, the SVM aims at finding the maximum margin hyperplane that separates the data points:

In a two-dimensional dataset, with linearly separable data points (as shown in the preceding figure), the maximum margin hyperplane would be a line that would maximize the distance between each of the classes.

The hyperplane could be expressed as a dot product of the set of input vectors  x and a vector normal to the hyperplane W:W.X=b, where b is the offset from the origin of the coordinate system.

To find the hyperplane, we solve the following optimization problem:

The constraint of our optimization problem effectively states that no point can cross the hyperplane if it does not belong to the class on that side of the hyperplane.

Linear SVM

Building a linear SVM classifier in Python is easy. There are multiple Python packages that can estimate a linear SVM but here, we decided to use MLPY (http://mlpy.sourceforge.net):

import pandas as pd
import numpy as np
import mlpy as ml

First, we load the necessary modules that we will use later, namely pandas (http://pandas.pydata.org), NumPy (http://www.numpy.org), and the aforementioned MLPY.

We use pandas to read the data (https://github.com/drabastomek/practicalDataAnalysisCookbook repository to download the data):

# the file name of the dataset
r_filename = 'Data/Chapter03/bank_contacts.csv'

# read the data
csv_read = pd.read_csv(r_filename)

The dataset that we use was described in S. Moro, P. Cortez, and P. Rita. A data-driven approach to Predict the Success of Bank Telemarketing. Decision Support Systems, Elsevier, 62:22-31, June 2014 and found here http://archive.ics.uci.edu/ml/datasets/Bank+Marketing. It consists of over 41.1k outbound marketing calls of a bank. Our aim is to classify these calls into two buckets: those that resulted in a credit application and those that did not.

Once the file was loaded, we split the data into training and testing datasets; we also keep the input and class indicator data separately. To this end, we use the split_dataset(...) method:

def split_data(data, y, x = 'All', test_size = 0.33):
    '''
        Method to split the data into training and testing
    '''
    import sys

    # dependent variable
    variables = {'y': y}

    # and all the independent
    if x == 'All':
        allColumns = list(data.columns)
        allColumns.remove(y)
        variables['x'] = allColumns
    else:
        if type(x) != list:
            print('The x parameter has to be a list...')
            sys.exit(1)
        else:
            variables['x'] = x

    # create a variable to flag the training sample
    data['train']  = np.random.rand(len(data)) < 
        (1 - test_size)

    # split the data into training and testing
    train_x = data[data.train] [variables['x']]
    train_y = data[data.train] [variables['y']]
    test_x  = data[~data.train][variables['x']]
    test_y  = data[~data.train][variables['y']]

    return train_x, train_y, test_x, test_y, variables['x']

We randomly set 1/3 of the dataset aside for testing purposes and use the remaining 2/3 for the training of the model:

# split the data into training and testing
train_x, train_y, 
test_x,  test_y, 
labels = hlp.split_data(
    csv_read,
    y = 'credit_application'
)

Once we read the data and split it into training and testing datasets, we can estimate the model:

# create the classifier object
svm = ml.LibSvm(svm_type='c_svc',
    kernel_type='linear', C=100.0)

# fit the data
svm.learn(train_x,train_y)

The svm_type parameter of the .LibSvm(...) method controls what algorithm to use to estimate the SVM. Here, we use the c_svc method—a C-support Vector Classifier. The method specifies how much you want to avoid misclassifying observations: the larger values of C parameter will shrink the margin for the hyperplane (theb) so that more of the observations are correctly classified. You can also specify nu_svc with a nu parameter that controls how much of your sample (at most) can be misclassified and how many of your observations (at least) can become support vectors.

Here, we estimate an SVM with a linear kernel, so let's talk about kernels.

Kernels

A kernel function K is effectively a function that computes a dot product between two n-dimensional vectors, K: R_n.R_n --> R. In other words, the kernel function takes two vectors and produces a scalar:

The linear kernel does not effectively transform the data into a higher dimensional space. This is not true for polynomial or Radial Basis Function (RBF) kernels that transform the input feature space into higher dimensions. In case of the polynomial kernel of degree d, the obtained feature space has (n+d/d) dimensions for the R_n dimensional input feature space.

As you can see, the number of additional dimensions can grow very quickly and this would pose significant problems in estimating the model if we would explicitly transform the data into higher dimensions. Thankfully, we do not have to do this as that's where the kernel trick comes into play.

The truth is that SVMs do not have to work explicitly in higher dimensions but can rather implicitly map the data to higher dimensions using pairwise inner products (instead of an explicit dot product) and then use it to find the maximum margin hyperplane. You can find a really good explanation of the kernel trick at http://www.eric-kim.net/eric-kim-net/posts/1/kernel_trick.html.

Back to our example

The .learn(...) method of the .LibSvm(...) object estimates the model.

Once the model is estimated, we can test how well it performs. First, we use the estimated model to predict the classes for the observations in the testing dataset:

predicted_l = svm.pred(test_x)

Next, we will use some of the scikit-learn methods to print the basic statistics for our model:

def printModelSummary(actual, predicted):
    '''
        Method to print out model summaries
    '''
    import sklearn.metrics as mt

    print('Overall accuracy of the model is {0:.2f} percent'
        .format(
            (actual == predicted).sum() / 
            len(actual) * 100))

    print('Classification report: n',
        mt.classification_report(actual, predicted))

    print('Confusion matrix: n',
        mt.confusion_matrix(actual, predicted))

    print('ROC: ', mt.roc_auc_score(actual, predicted))

First, we calculate the overall accuracy of the model expressed as a ratio of properly classified observations to the total number of observations in the testing sample. Next, we print the classification report:

The precision is the model's ability to avoid classifying an observation as positive when it is not. It is a ratio of true positives to the overall number of positively classified records. The overall precision score is a weighted average of the individual precision scores where the weight is the support. The support is the total number of actual observations in each class.

The total precision for our model is not too bad—89 out of 100. However, when we look at the precision to classify the true positives, the situation is not as good—only 63 out of 100 were properly classified.

Recall can be viewed as the model's capacity to find all the positive samples. It is a ratio of true positives to the sum of true positives and false negatives. The recall for the class 0.0 is almost perfect but for class 1.0, it looks really bad. This might be a problem with the fact that our sample is not balanced, but it is more likely that the features we use to classify the data do not really capture the differences between the two groups.

The f1-score is effectively a weighted amalgam of the precision and recall: it is a ratio of twice the product of precision and recall to their sum. In one measure, it shows whether the model performs well or not.

At the general level, the model does not perform badly but when looked at the model's ability to classify the true signal, it fails gravely. It is a perfect example why judging the model at the general level might be misleading when dealing with samples that are heavily unbalanced.

RBF kernel SVM

Given that the linear kernel performed poorly, our dataset might not be linearly separable. Thus, let's try the RBF kernel.

The RBF kernel is given as K(x,y)=e ||x-y||^2/2a2, where ||x-y||² is a Euclidean distance between the two vectors, x and y, and σ is a free parameter. The value of RBF equals to 1 when x=y and gradually falls to 0 when the distance approaches infinity.

To fit an RBF version of our model, we can specify our svm object as follows:

svm = ml.LibSvm(svm_type='c_svc', kernel_type='rbf',
        gamma=0.1, C=1.0)

The gamma parameter here specifies how far the influence of a single support vector reaches. Visually, you can investigate the relationship between gamma and C parameters at http://scikit-learn.org/stable/auto_examples/svm/plot_rbf_parameters.html.

The rest of the code for the model estimation follows in a similar fashion as with the linear kernel and we obtain the following results:

The results are even worse than the linear kernel as the precision and recall were lost across the board. The SVM with the RBF kernel performed worse when classifying calls that resulted in applying for the credit card and those that did not.

Summary

In this article, we saw that the problem is not with the model but rather, the dataset that we use does not explain the variance sufficiently. This requires going back to the drawing board and selecting other features.

Resources for Article:

Further resources on this subject:

• Push your data to the Web [article]
• Transferring Data from MS Access 2003 to SQL Server 2008 [article]
• Exporting data from MS Access 2003 to MySQL [article]