Using Latent Semantic Analysis for text analytics with Spark 2.0
In this recipe, we will explore LSA utilizing a data dump of articles from Wikipedia. LSA translates into analyzing a corpus of documents to find hidden meaning or concepts in those documents.
In the first recipe of this chapter, we covered the basics of the TF (that is, term frequency) technique. In this recipe, we use HashingTF for calculating TF and use IDF to fit a model into the calculated TF. At its core, LSA uses singular value decomposition (SVD) on the term frequency document to reduce dimensionality and therefore extract the most important concepts. There are other cleanup steps that we need to do (for example, stop words and stemming) that will clean up the bag of words before we start analyzing it.
How to do it...
- Start a new project in IntelliJ or in an IDE of your choice. Make sure the necessary JAR files are included.
- The package statement for the recipe is as follows:
package spark.ml.cookbook.chapter12- Import the...
