Using Spark Notebooks for quick iteration of ideas

In this section, we will answer the following questions:

• What are Spark Notebooks?
• How do you start Spark Notebooks?
• How do you use Spark Notebooks?

Let's start with setting up a Jupyter Notebook-like environment for Spark. Spark Notebook is just an interactive and reactive data science environment that uses Scala and Spark.

If we view the GitHub page (https://github.com/spark-notebook/spark-notebook), we can see that what the Notebooks do is actually very straightforward, as shown in the following screenshot:

If we look at a Spark Notebook, we can see that they look very much like what Python developers use, which is Jupyter Notebooks. You have a text box allowing you to enter some code, and then you execute the code below the text box, which is similar to a Notebook format. This allows us to perform a reproducible analysis...