Using serialization to improve performance

Serialization plays an important part in distributed computing. There are two persistence (storage) levels that support serializing RDDs:

• MEMORY_ONLY_SER: This stores RDDs as serialized objects. It will create one byte array per partition.
• MEMORY_AND_DISK_SER: This is similar to MEMORY_ONLY_SER, but it spills partitions that do not fit in the memory to disk.

How to do it...

The following are the steps to add appropriate persistence levels:

1. Start the Spark shell:

        $ spark-shell

2. Import the StorageLevel object as enumeration of persistence levels and the implicits associated with it:

        scala> import org.apache.spark.storage.StorageLevel._

3. Create a dataset:

        scala> val words = spark.read...