Technical requirements

You can also find the code for this chapter in the GitHub repository here: https://github.com/PacktPublishing/Data-Ingestion-with-Python-Cookbook.

Using Jupyter Notebook is not mandatory but can help you see how the code works interactively. Since we will execute Python and PySpark code, it can help us understand the scripts better. Once you have it installed, you can execute Jupyter using the following line:

$ jupyter Notebook

It is recommended to create a separate folder to store the Python files or Notebooks we will cover in this chapter; however, feel free to organize the files in the best way that fits you.

In this chapter, all recipes will need a SparkSession instance initialized, and you can use the same session for all of them. You can use the following code to create your session:

from pyspark.sql import SparkSession
spark = SparkSession.builder \
      .master("local[1]") \
    ...