Solving simple text mining problems with Apache Spark

According to the Apache Spark website, Spark runs programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk. Generally speaking, Apache Spark is an open-source cluster-computing framework. Its processing engine provides good speed with ease of use, and it offers sophisticated analytics to data scientists.

In this recipe, we will demonstrate how you can use Apache Spark to solve very simple data problems. Of course, the data problems are merely dummy problems and not real-world problems, but this can be a starting point for you to understand intuitively the use of Apache Spark for using it on a large scale.

Getting ready

1. In Eclipse, create a new Maven project. The author had Eclipse Mars set up. To do so, go to File. Then select New and Other...:

   

2. Expand Maven from the wizard and select Maven Project. Click on Next until you reach the window where Eclipse prompts you to provide an Artifact Id. Type in mlib as the Artifact...