Extending Spark
As an open source platform, Apache Spark MLlib can be easily customized, and extended to address specific problems related to requiring the implementation of new statistics, mathematical, or machine learning solutions.
The purpose of this section is to create a Kullback-Leibler divergence as a Spark evaluator. The implementation follows two steps:
- Describe and implement the Kullback-Leibler divergence using Spark 2.0
- Convert the class into a Spark evaluator
Kullback-Leibler divergence
The Kullback-Leibler has been briefly introduced in Chapter 5, Dimension Reduction – Divergences in the context of evaluating the similarity of the frequencies distribution between two datasets. The Kullback-Leibler divergence is a concept borrowed from information theory and is commonly associated with Information Gain. The Kullback-Leibler divergence is also known as the relative entropy or information divergence between two distributions. It measures the dissimilarity of the distribution...
