Using PySpark to read JSON files

In the Reading a JSON file recipe, we saw that JSON files are widely used to transport and share data between applications, and we saw how to read a JSON file using simple Python code.

However, with the increase in data size and sharing, using only Python to process a high volume of data can lead to performance or resilience issues. That’s why, for this type of scenario, it is highly recommended to use PySpark to read and process JSON files. As you might expect, PySpark comes with a straightforward reading solution.

In this recipe, we will cover how to read a JSON file with PySpark, the common associated issues, and how to solve them.

Getting ready

As in the previous recipe, Reading a JSON file, we are going to use the GitHub Events JSON file. Also, the use of Jupyter Notebook is optional.

How to do it…

Here are the steps for this recipe:

1. We first create the SparkSession:

   spark = .builder \
     &...