Technical requirements

Like Chapter 6, in this chapter too, some recipes will need SparkSession initialized, and you can use the same session for all of them. You can use the following code to create your session:

from pyspark.sql import SparkSession
spark = SparkSession.builder \
      .master("local[1]") \
      .appName("chapter7_analytical_data") \
      .config("spark.executor.memory", '3g') \
      .config("spark.executor.cores", '2') \
      .config("spark.cores.max", '2') \
      .getOrCreate()

Note

A WARN message as output is expected in some cases, especially if you are using WSL on Windows, so you don’t need to worry if you receive one.

You can also find the code from this chapter in its GitHub repository...