Introducing Spark ML tools and utilities

In the following sections, we will explore various and that Spark ML offers to select features and create superior ML models easily and efficiently.

Using Principal Component Analysis to select features

As mentioned earlier, we can derive features using Principal Component Analysis (PCA) on the data. This approach depends on the problem, so it is imperative to have a good understanding about the domain.

This exercise typically requires creativity and common sense to a set of features may be relevant to the problem. A more extensive exploratory data analysis is typically required to help understand the data better and/or to identify patterns that lead to a good set of features.

PCA is a statistical procedure that converts a set of potentially correlated variables into a, typically, reduced set of linearly uncorrelated variables. The resulting set of uncorrelated variables are called principal components. A PCA class trains a model to project vectors...