Creating SparseVector and setup with Spark

In this recipe, we several types of SparseVector creation. As the length of the vector increases (millions) and the density remains low (few non-zero members), then sparse representation more and more advantageous over the DenseVector.

How to do it...

1. Start a new project in IntelliJ or in an IDE of your choice. Make sure that the necessary JAR files are included.
2. Import the necessary packages for vector and matrix manipulation:

import org.apache.spark.sql.{SparkSession}
import org.apache.spark.mllib.linalg._
import breeze.linalg.{DenseVector => BreezeVector}
import Array._
import org.apache.spark.mllib.linalg.SparseVector

3. Set up the Spark context and application parameters so Spark can run. See the first recipe in this chapter for more details and variations:

val spark = SparkSession
 .builder
 .master("local[*]")
 .appName("myVectorMatrix")
 .config("spark.sql.warehouse.dir", ".")
 .getOrCreate()

4. Here we look at creating a ML SparseVector that corresponds...