Support Vector Machines

SVMs are classification algorithms based on a simple and intuitive idea, backed by some complex and innovative mathematics. SVMs perform classification between two classes (although we can extend it to more classes using various meta-algorithms), by simply drawing a separating line between the two (or a hyperplane in higher-dimensions). The intuitive idea is to choose the best line of separation, rather than just any specific line.

Suppose that our two classes can be separated by a line such that any points above the line belong to one class and any below the line belong to the other class. SVMs find this line and use it for prediction, much the same way as linear regression works. SVMs, however, find the best line for separating the dataset. In the following figure, we have three lines that separate the dataset: blue, black, and green. Which would you say is the best option?

Intuitively, a person would normally choose the blue line as the best option, as this separates...