Introduction

In this chapter, you will see three key technologies used in Big Data framework, which are extremely useful for data scientists: Apache Mahout, Apache Spark, and its machine learning library named MLib.

We will start our chapter with Apache Mahout--a scalable or distributed machine learning platform for classification, regression, clustering, and collaborative filtering tasks. Mahout started as a machine learning workbench that works only on Hadoop MapReduce but eventually selected Apache Spark as its platform.

Apache Spark is a framework that brings in parallelization in Big Data processing and has similarity with MapReduce as it also distributes data across clusters. But one key difference between Spark and MapReduce is the prior attempts to keep things in memory as much as possible while the latter writes and reads continuously from the disks. Therefore, Spark is much faster than MapReduce. We will see how you, as a data scientist, can use Spark to do simple text-mining related...