MLlib library

MLlib is a scalable machine learning library built on top of Spark. The machine learning library is composed of two distinct packages, which are [17:03]:

1. org.apache.spark.mllib: RDD-based library of some common machine learning algorithms. This package will be deprecated in future releases.
2. org.apache.spark.ml: Library of machine learning algorithms that leverages datasets and data frames structures. The package supports tasks pipeline and stages that are described and illustrated in the next section.

Overview

The main components of the MLlib package are as follows:

• Classification algorithms, including logistic regression, Naïve Bayes, and support vector machines
• Clustering and unsupervised learning techniques such as K-means
• L₁ and L₂ regularization
• Optimization techniques such as gradient descent, logistic gradient and stochastic gradient descent, and L-BFGS
• Linear algebra such as singular value decomposition
• Data generator for K-means, logistic regression, and support vector...