Transforming RDDs with the zip() API
In this recipe we explore the zip() function. For those of us working in Python or Scala, zip() is a familiar method that lets you pair items before applying an inline function. Using Spark, it can be used to facilitate RDD arithmetic between pairs. Conceptually, it combines the two RDDs in such a way that each member of one RDD is paired with the second RDD that occupies the same position (that is, it lines up the two RDDs and makes pairs out of the members).
How to do it...
- Start a new project in IntelliJ or in an IDE of your choice. Make sure the necessary JAR files are included.
- Set up the package location where the program will reside
package spark.ml.cookbook.chapter3
- Import the necessary packages
import org.apache.spark.sql.SparkSession
- Import the packages for setting up the logging level for
log4j. This step is optional, but we highly recommend it (change the level appropriately as you move through the development cycle).
import org.apache.log4j...
