Chapter 5. Apache SparkML

So now that you've learned a lot about MLlib, why another ML API? First of all, it is a common task in data science to work with multiple frameworks and ML libraries as there are always advantages and disadvantages; mostly, it is a trade-off between performance and functionality. R, for instance, is the king when it comes to functionality--there exist more than 6000 R add-on packages. However, R is also one of the slowest execution environments for data science. SparkML, on the other hand, currently has relatively limited functionality but is one of the fastest libraries. Why is this so? This brings us to the second reason why SparkML exists.

The duality between RDD on the one hand and DataFrames and Datasets on the other is like a red thread in this book and doesn't stop influencing the machine learning chapters. As MLlib is designed to work on top of RDDs, SparkML works on top of DataFrames and Datasets, therefore making use of all the new performance benefits...