Clustering techniques

Clustering is an unsupervised learning technique where there is no response variable to supervise the model. The idea is to cluster the data points that have some level of similarity. Apart from exploratory data analysis, it is also used as a part of a supervised pipeline where classifiers or regressors can be built on the distinct clusters. There are a bunch of clustering techniques available. Let us look into a few important ones that are supported by Spark.

K-means clustering

K-means is one of the most common clustering techniques. The k-means problem is to find cluster centers that minimize the intra-class variance, that is, the sum of squared distances from each data point being clustered to its cluster center (the center that is closest to it). You have to specify in advance the number of clusters you want in the dataset.

Since it uses the Euclidian distance measure to find the differences between the data points, the features need to be scaled to a comparable unit...