In Chapter 4, Obtaining, Processing, and Preparing Data with Spark, we covered various topics related to feature extraction and data processing, including the basics of extracting features from text data. In this chapter, we will introduce more advanced text processing techniques available in Spark ML to work with large-scale text datasets.
In this chapter, we will:
- Work through detailed examples that illustrate data processing, feature extraction, and the modeling pipeline, as they relate to text data
- Evaluate the similarity between two documents based on the words in the documents
- Use the extracted text features as inputs for a classification model
- Cover a recent development in natural language processing to model words themselves as vectors and illustrate the use of Spark's Word2Vec model to evaluate the similarity between two words, based on their meaning
We will look...
