NGrams

NGrams are word combinations created as sequences of words. N stands for the number of words in the sequence. For example, 2-gram is two words together, 3-gram is three words together. setN() is used to specify the value of N.

In order to generate NGrams, you need to import the package:

import org.apache.spark.ml.feature.NGram

First, you need to initialize an NGram generator specifying the input column and the output column. Here, we are choosing the filtered words column created by the StopWordsRemover and generating an output column for the filtered words after removal of stop words:

scala> val ngram = new NGram().setN(2).setInputCol("filteredWords").setOutputCol("ngrams")
ngram: org.apache.spark.ml.feature.NGram = ngram_e7a3d3ab6115

Next, invoking the transform() function on the input dataset yields an output dataset:

scala> val nGramDF = ngram...