Reusable ML pipelines

ML pipelines have been introduced in Apache Spark 1.4.0. An ML pipeline is a sequence of tasks that can be used to cleanse, filter, train, classify observations, detect anomalies, generate, validate models, and predict outcomes [17:04].

Contrary to the MLlib package classes that rely on RDDs, ML pipeline uses data frame or datasets as input and output of tasks.

Note

Data frame versus Dataset

The class Dataset was introduced in Spark 2.0. Dataset instances are typed (that is, Dataset[T]) while data frames are untyped.

This section is a very brief overview of ML pipelines.

The key ingredients of an ML pipeline are [17:05]:

• Transformers are algorithms that can transform one data frame into another data frame. Transformers are stateless.
• Estimators are algorithms that can fit on a data frame to produce a Transformer (that is, Estimator.fit).
• Pipelines are estimators that weave or chain multiple transformers and estimators together to specify an ML workflow.
• Pipeline stages are...