Deep learning is the focused study of machine learning algorithms that deploy neural networks as their main method of learning. Deep learning has exploded onto the scene just within the last couple of years. Microsoft, Google, Facebook, Amazon, Apple, Tesla and many other companies are all utilizing deep learning models in their apps, websites, and products. At the same exact time, Spark, an in-memory compute engine running on top of big data sources, has made it easy to process volumes of information at record speeds and ease. In fact, Spark has now become the leading big data development tool for data engineers, machine learning engineers, and data scientists.

Since deep learning models perform better with more data, the synergy between Spark and deep learning allowed for a perfect marriage. Almost as important as the code used to execute deep learning algorithms is the work environment that enables optimal development. Many talented minds are eager to develop neural networks to help answer important questions in their research. Unfortunately, one of the greatest barriers to the development of deep learning models is access to the necessary technical resources required to learn on big data. The purpose of this chapter is to create an ideal virtual development environment for deep learning on Spark.

Spark can be set up for all types of operating systems, whether they reside on-premise or in the cloud. For our purposes, Spark will be installed on a Linux-based virtual machine with Ubuntu as the operating system. There are several advantages to using Ubuntu as the go-to virtual machine, not least of which is cost. Since they are based on open source software, Ubuntu operating systems are free to use and do not require licensing. Cost is always a consideration and one of the main goals of this publication is to minimize the financial footprint required to get started with deep learning on top of a Spark framework.

There are some minimum recommendations required for downloading the image file:

• Minimum of 2 GHz dual-core processor
• Minimum of 2 GB system memory
• Minimum of 25 GB of free hard drive space

Follow the steps in the recipe to download an Ubuntu Desktop image:

1. In order to create a virtual machine of Ubuntu Desktop, it is necessary to first download the file from the official website: https://www.ubuntu.com/download/desktop.

2. As of this writing, Ubuntu Desktop 16.04.3 is the most recent available version for download.

3. Access the following file in a .iso format once the download is complete:

   ubuntu-16.04.3-desktop-amd64.iso

Virtual environments provide an optimal development workspace by isolating the relationship to the physical or host machine. Developers may be using all types of machines for their host environments such as a MacBook running macOS, a Microsoft Surface running Windows or even a virtual machine on the cloud with Microsoft Azure or AWS; however, to ensure consistency within the output of the code executed, a virtual environment within Ubuntu Desktop will be deployed that can be used and shared among a wide variety of host platforms.

There are several options for desktop virtualization software, depending on whether the host environment is on a Windows or a macOS. There are two common software applications for virtualization when using macOS:

• VMWare Fusion
• Parallels

To learn more about Ubuntu Desktop, you can visit https://www.ubuntu.com/desktop.

This section will focus on building a virtual machine using an Ubuntu operating system with VMWare Fusion.

A previous installation of VMWare Fusion is required on your system. If you do not currently have this, you can download a trial version from the following website:

https://www.vmware.com/products/fusion/fusion-evaluation.html

Follow the steps in the recipe to configure Ubuntu with VMWare Fusion on macOS:

1. Once VMWare Fusion is up and running, click on the + button on the upper-left-hand side to begin the configuration process and select New..., as seen in the following screenshot:

2. Once the selection has been made, select the option to Install from Disk or Image, as seen in the following screenshot:

3. Select the operating system's iso file that was downloaded from the Ubuntu Desktop website, as seen in the following screenshot:

4. The next step will ask whether you want to choose Linux Easy Install. It is recommended to do so, as well as to incorporate a Display Name/Password combination for the Ubuntu environment, as seen in the following screenshot:

5. The configuration process is almost complete. A Virtual Machine Summary is displayed with the option to Customize Settings to increase the Memory and Hard Disk, as seen in the following screenshot:

6. Anywhere from 20 to 40 GB hard disk space is sufficient for the virtual machine; however, bumping up the memory to either 2 GB or even 4 GB will assist with the performance of the virtual machine when executing Spark code in later chapters. Update the memory by selecting Processors and Memory under the Settings of the virtual machine and increasing the Memory to the desired amount, as seen in the following screenshot:

The setup allows for manual configuration of the settings necessary to get Ubuntu Desktop up and running successfully on VMWare Fusion. The memory and hard drive storage can be increased or decreased based on the needs and availability of the host machine.

All that is remaining is to fire up the virtual machine for the first time, which initiates the installation process of the system onto the virtual machine. Once all the setup is complete and the user has logged in, the Ubuntu virtual machine should be available for development, as seen in the following screenshot:

Aside from VMWare Fusion, there is also another product that offers similar functionality on a Mac. It is called Parallels Desktop for Mac. To learn more about VMWare and Parallels, and decide which program is a better fit for your development, visit the following websites:

• https://www.vmware.com/products/fusion.html to download and install VMWare Fusion for Mac
• https://parallels.com to download and install the Parallels Desktop for Mac

Unlike with macOS, there are several options to virtualize systems within Windows. This mainly has to do with the fact that virtualization on Windows is very common as most developers are using Windows as their host environment and need virtual environments for testing purposes without affecting any of the dependencies that rely on Windows.

VirtualBox from Oracle is a common virtualization product and is free to use. Oracle VirtualBox provides a straightforward process to get an Ubuntu Desktop virtual machine up and running on top of a Windows environment.

Follow the steps in this recipe to configure Ubuntu with VirtualBox on Windows:

1. Initiate an Oracle VM VirtualBox Manager. Next, create a new virtual machine by selecting the New icon and specify the Name, Type, and Version of the machine, as seen in the following screenshot:

2. Select Expert Mode as several of the configuration steps will get consolidated, as seen in the following screenshot:

Ideal memory size should be set to at least 2048 MB, or preferably 4096 MB, depending on the resources available on the host machine.

3. Additionally, set an optimal hard disk size for an Ubuntu virtual machine performing deep learning algorithms to at least 20 GB, if not more, as seen in the following screenshot:

4. Point the virtual machine manager to the start-up disk location where the Ubuntu iso file was downloaded to and then Start the creation process, as seen in the following screenshot:

5. After allotting some time for the installation, select the Start icon to complete the virtual machine and get it ready for development as seen in the following screenshot:

The setup allows for manual configuration of the settings necessary to get Ubuntu Desktop up and running successfully on Oracle VirtualBox. As was the case with VMWare Fusion, the memory and hard drive storage can be increased or decreased based on the needs and availability of the host machine.

Please note that some machines that run Microsoft Windows are not set up by default for virtualization and users may receive an initial error indicating the VT-x is not enabled. This can be reversed and virtualization may be enabled in the BIOS during a reboot.

To learn more about Oracle VirtualBox and decide whether or not it is a good fit, visit the following website and select Windows hosts to begin the download process: https://www.virtualbox.org/wiki/Downloads.

Previously, we saw how Ubuntu Desktop could be set up locally using VMWare Fusion. In this section, we will learn how to do the same on Google Cloud Platform.

The only requirement is a Google account username. Begin by logging in to your Google Cloud Platform using your Google account. Google provides a free 12-month subscription with $300 credited to your account. The setup will ask for your bank details; however, Google will not charge you for anything without explicitly letting you know first. Go ahead and verify your bank account and you are good to go.

Follow the steps in the recipe to configure Ubuntu Desktop for Google Cloud Platform:

1. Once logged in to your Google Cloud Platform, access a dashboard that looks like the one in the following screenshot:

2. First, click on the product services button in the top-left-hand corner of your screen. In the drop-down menu, under Compute, click on VM instances, as shown in the following screenshot:

3. Create a new instance and name it. We are naming it ubuntuvm1 in our case. Google Cloud automatically creates a project while launching an instance and the instance will be launched under a project ID. The project may be renamed if required.

4. After clicking on Create Instance, select the zone/area you are located in.
5. Select Ubuntu 16.04LTS under the boot disk as this is the operating system that will be installed in the cloud. Please note that LTS stands for version, and will have long-term support from Ubuntu’s developers.
6. Next, under the boot disk options, select SSD persistent disk and increase the size to 50 GB for some added storage space for the instance, as shown in the following screenshot:

7. Next, set Access scopes to Allow full access to all Cloud APIs.
8. Under firewall, please check to allow HTTP traffic as well as allow HTTPS traffic, as shown in the following screenshot:

9. Once the instance is configured as shown in this section, go ahead and create the instance by clicking on the Create button.

10.  Click on the newly created instance. From the drop-down menu, click on open in browser window, as shown in the following screenshot:

11. You will see that Google opens up a shell/terminal in a new window, as shown in the following screenshot:

12. Once the shell is open, you should have a window that looks like the following screenshot:

13. Type the following commands in the Google cloud shell:

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install gnome-shell
$ sudo apt-get install ubuntu-gnome-desktop
$ sudo apt-get install autocutsel
$ sudo apt-get install gnome-core
$ sudo apt-get install gnome-panel
$ sudo apt-get install gnome-themes-standard

14. When presented with a prompt to continue or not, type y and select ENTER, as shown in the following screenshot:

15. Once done with the preceding steps, type the following commands to set up the vncserver and allow connections to the local shell:

$ sudo apt-get install tightvncserver
$ touch ~/.Xresources

16. Next, launch the server by typing the following command:

$ tightvncserver

17. This will prompt you to enter a password, which will later be used to log in to the Ubuntu Desktop virtual machine. This password is limited to eight characters and needs to be set and verified, as shown in the following screenshot:

18. A startup script is automatically generated by the shell, as shown in the following screenshot. This startup script can be accessed and edited by copying and pasting its PATH in the following manner:

19. In our case, the command to view and edit the script is:

:~$ vim /home/amrith2kmeanmachine/.vnc/xstartup

This PATH may be different in each case. Ensure you set the right PATH. The vim command opens up the script in the text editor on a Mac.

20. After typing the vim command, the screen with the startup script should look something like this screenshot:

21. Type i to enter INSERT mode. Next, delete all the text in the startup script. It should then look like the following screenshot:

22. Copy paste the following code into the startup script:

#!/bin/sh
autocutsel -fork
xrdb $HOME/.Xresources
xsetroot -solid grey
export XKL_XMODMAP_DISABLE=1
export XDG_CURRENT_DESKTOP="GNOME-Flashback:Unity"
export XDG_MENU_PREFIX="gnome-flashback-"
unset DBUS_SESSION_BUS_ADDRESS
gnome-session --session=gnome-flashback-metacity --disable-acceleration-check --debug &

23. The script should appear in the editor, as seen in the following screenshot:

24. Press Esc to exit out of INSERT mode and type :wq to write and quit the file.
25. Once the startup script has been configured, type the following command in the Google shell to kill the server and save the changes:

$ vncserver -kill :1

26. This command should produce a process ID that looks like the one in the following screenshot:

27. Start the server again by typing the following command:

$ vncserver -geometry 1024x640

The next series of steps will focus on securing the shell tunnel into the Google Cloud instance from the local host. Before typing anything on the local shell/terminal, ensure that Google Cloud is installed. If not already installed, do so by following the instructions in this quick-start guide located at the following website:

https://cloud.google.com/sdk/docs/quickstart-mac-os-x

28. Once Google Cloud is installed, open up the terminal on your machine and type the following commands to connect to the Google Cloud compute instance:

$ gcloud compute ssh \
YOUR INSTANCE NAME HERE \
--project YOUR PROJECT NAME HERE \
--zone YOUR TIMEZONE HERE \
--ssh-flag "-L 5901:localhost:5901"

29. Ensure that the instance name, project ID, and zone are specified correctly in the preceding commands. On pressing ENTER, the output on the local shell changes to what is shown in the following screenshot:

30. Once you see the name of your instance followed by ":~$", it means that a connection has successfully been established between the local host/laptop and the Google Cloud instance. After successfully SSHing into the instance, we require software called VNC Viewer to view and interact with the Ubuntu Desktop that has now been successfully set up on the Google Cloud Compute engine. The following few steps will discuss how this is achieved.

31. VNC Viewer may be downloaded using the following link:

https://www.realvnc.com/en/connect/download/viewer/

32. Once installed, click to open VNC Viewer and in the search bar, type in localhost::5901, as shown in the following screenshot:

33. Next, click on continue when prompted with the following screen:

34. This will prompt you to enter your password for the virtual machine. Enter the password that you set earlier while launching the tightvncserver command for the first time, as shown in the following screenshot:

35. You will finally be taken into the desktop of your Ubuntu virtual machine on Google Cloud Compute. Your Ubuntu Desktop screen must now look something like the following screenshot when viewed on VNC Viewer:

You have now successfully set up VNC Viewer for interactions with the Ubuntu virtual machine/desktop. Anytime the Google Cloud instance is not in use, it is recommended to suspend or shut down the instance so that additional costs are not being incurred. The cloud approach is optimal for developers who may not have access to physical resources with high memory and storage.

While we discussed Google Cloud as a cloud option for Spark,  it is possible to leverage Spark on the following cloud platforms as well:

• Microsoft Azure
• Amazon Web Services

In order to learn more about Google Cloud Platform and sign up for a free subscription, visit the following website:

https://cloud.google.com/

Before Spark can get up and running, there are some necessary prerequisites that need to be installed on a newly minted Ubuntu Desktop. This section will focus on installing and configuring the following on Ubuntu Desktop:

• Java 8 or higher
• Anaconda
• Spark

The only requirement for this section is having administrative rights to install applications onto the Ubuntu Desktop.

This section walks through the steps in the recipe to install Python 3, Anaconda, and Spark on Ubuntu Desktop:

1. Install Java on Ubuntu through the terminal application, which can be found by searching for the app and then locking it to the launcher on the left-hand side, as seen in the following screenshot:

2. Perform an initial test for Java on the virtual machine by executing the following command at the terminal:

java -version

3. Execute the following four commands at the terminal to install Java:

sudo apt-get install software-properties-common 
$ sudo add-apt-repository ppa:webupd8team/java
$ sudo apt-get update
$ sudo apt-get install oracle-java8-installer

4. After accepting the necessary license agreements for Oracle, perform a secondary test of Java on the virtual machine by executing java -version once again in the terminal. A successful installation for Java will display the following outcome in the terminal:

$ java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

5. Next, install the most recent version of Anaconda. Current versions of Ubuntu Desktop come preinstalled with Python. While it is convenient that Python comes preinstalled with Ubuntu, the installed version is for Python 2.7, as seen in the following output:

$ python --version
Python 2.7.12

6. The current version of Anaconda is v4.4 and the current version of Python 3 is v3.6. Once downloaded, view the Anaconda installation file by accessing the Downloads folder using the following command:

$ cd Downloads/
~/Downloads$ ls
Anaconda3-4.4.0-Linux-x86_64.sh

7. Once in the Downloads folder, initiate the installation for Anaconda by executing the following command:

~/Downloads$ bash Anaconda3-4.4.0-Linux-x86_64.sh 
Welcome to Anaconda3 4.4.0 (by Continuum Analytics, Inc.)
In order to continue the installation process, please review the license agreement.
Please, press ENTER to continue

8. Once the Anaconda installation is complete, restart the Terminal application to confirm that Python 3 is now the default Python environment through Anaconda by executing python --version in the terminal:

$ python --version
Python 3.6.1 :: Anaconda 4.4.0 (64-bit)

9. The Python 2 version is still available under Linux, but will require an explicit call when executing a script, as seen in the following command:

~$ python2 --version
Python 2.7.12

10. Visit the following website to begin the Spark download and installation process:

https://spark.apache.org/downloads.html

11. Select the download link. The following file will be downloaded to the Downloads folder in Ubuntu:

spark-2.2.0-bin-hadoop2.7.tgz

12. View the file at the terminal level by executing the following commands:

$ cd Downloads/
~/Downloads$ ls
spark-2.2.0-bin-hadoop2.7.tgz

13. Extract the tgz file by executing the following command:

~/Downloads$ tar -zxvf spark-2.2.0-bin-hadoop2.7.tgz

14. Another look at the Downloads directory using ls shows both the tgz file and the extracted folder:

~/Downloads$ ls
spark-2.2.0-bin-hadoop2.7 spark-2.2.0-bin-hadoop2.7.tgz

15. Move the extracted folder from the Downloads folder to the Home folder by executing the following command:

~/Downloads$ mv spark-2.2.0-bin-hadoop2.7 ~/
~/Downloads$ ls
spark-2.2.0-bin-hadoop2.7.tgz
~/Downloads$ cd
~$ ls
anaconda3 Downloads Pictures Templates
Desktop examples.desktop Public Videos
Documents Music spark-2.2.0-bin-hadoop2.7

16. Now, the spark-2.2.0-bin-hadoop2.7 folder has been moved to the Home folder, which can be viewed when selecting the Files icon on the left-hand side toolbar, as seen in the following screenshot:

17. Spark is now installed. Initiate Spark from the terminal by executing the following script at the terminal level:

~$ cd ~/spark-2.2.0-bin-hadoop2.7/
~/spark-2.2.0-bin-hadoop2.7$ ./bin/pyspark

18. Perform a final test to ensure Spark is up and running at the terminal by executing the following command to ensure that the SparkContext is driving the cluster in the local environment:

>>> sc
<SparkContext master=local[*] appName=PySparkShell>

This section explains the reasoning behind the installation process for Python, Anaconda, and Spark.

1. Spark runs on the Java virtual machine (JVM), the Java Software Development Kit (SDK) is a prerequisite installation for Spark to run on an Ubuntu virtual machine.

2. Ubuntu recommends the sudo apt install method for Java as it ensures that packages downloaded are up to date. 
3. Please note that if Java is not currently installed, the output in the terminal will show the following message:

The program 'java' can be found in the following packages:
* default-jre
* gcj-5-jre-headless
* openjdk-8-jre-headless
* gcj-4.8-jre-headless
* gcj-4.9-jre-headless
* openjdk-9-jre-headless
Try: sudo apt install <selected package>

4. While Python 2 is fine, it is considered legacy Python. Python 2 is facing an end of life date in 2020; therefore, it is recommended that all new Python development be performed with Python 3, as will be the case in this publication. Up until recently, Spark was only available with Python 2. That is no longer the case. Spark works with both Python 2 and 3. A convenient way to install Python 3, as well as many dependencies and libraries, is through Anaconda. Anaconda is a free and open source distribution of Python, as well as R. Anaconda manages the installation and maintenance of many of the most common packages used in Python for data science-related tasks.

5. During the installation process for Anaconda, it is important to confirm the following conditions: 

   • Anaconda is installed in the /home/username/Anaconda3 location
   • The Anaconda installer prepends the Anaconda3 install location to a PATH in /home/username/.bashrc

6. After Anaconda has been installed, download Spark. Unlike Python, Spark does not come preinstalled on Ubuntu and therefore, will need to be downloaded and installed.

7. For the purposes of development with deep learning, the following preferences will be selected for Spark:

   • Spark release: 2.2.0 (Jul 11 2017)
   • Package type: Prebuilt for Apache Hadoop 2.7 and later
   • Download type: Direct download
8. Once Spark has been successfully installed, the output from executing Spark at the command line should look something similar to that shown in the following screenshot:

9. Two important features to note when initializing Spark are that it is under the Python 3.6.1 | Anaconda 4.4.0 (64-bit) | framework and that the Spark logo is version 2.2.0.
10. Congratulations! Spark is successfully installed on the local Ubuntu virtual machine. But, not everything is complete. Spark development is best when Spark code can be executed within a Jupyter notebook, especially for deep learning. Thankfully, Jupyter has been installed with the Anaconda distribution performed earlier in this section.

You may be asking why we did not just use pip install pyspark to use Spark in Python. Previous versions of Spark required going through the installation process that we did in this section. Future versions of Spark, starting with 2.2.0 will begin to allow installation directly through the pip approach. We used the full installation method in this section to ensure that you will be able to get Spark installed and fully-integrated, in case you are using an earlier version of Spark.

To learn more about Jupyter notebooks and their integration with Python, visit the following website:

http://jupyter.org

To learn more about Anaconda and download a version for Linux, visit the following website: 

https://www.anaconda.com/download/.

When learning Python for the first time, it is useful to use Jupyter notebooks as an interactive developing environment (IDE). This is one of the main reasons why Anaconda is so powerful. It fully integrates all of the dependencies between Python and Jupyter notebooks. The same can be done with PySpark and Jupyter notebooks. While Spark is written in Scala, PySpark allows for the translation of code to occur within Python instead.

Most of the work in this section will just require accessing the .bashrc script from the terminal.

PySpark is not configured to work within Jupyter notebooks by default, but a slight tweak of the .bashrc script can remedy this issue. We will walk through these steps in this section:

1. Access the .bashrc script by executing the following command:

$ nano .bashrc

2. Scrolling all the way to the end of the script should reveal the last command modified, which should be the PATH set by Anaconda during the installation earlier in the previous section. The PATH should appear as seen in the following:

# added by Anaconda3 4.4.0 installer
export PATH="/home/asherif844/anaconda3/bin:$PATH"

3. Underneath, the PATH added by the Anaconda installer can include a custom function that helps communicate the Spark installation with the Jupyter notebook installation from Anaconda3. For the purposes of this chapter and remaining chapters, we will name that function sparknotebook. The configuration should appear as the following for sparknotebook():

function sparknotebook()
{
export SPARK_HOME=/home/asherif844/spark-2.2.0-bin-hadoop2.7
export PYSPARK_PYTHON=python3
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"
$SPARK_HOME/bin/pyspark
}

4. The updated .bashrc script should look like the following once saved:

5. Save and exit from the .bashrc file. It is recommended to communicate that the .bashrc file has been updated by executing the following command and restarting the terminal application:

$ source .bashrc

Our goal in this section is to integrate Spark directly into a Jupyter notebook so that we are not doing our development at the terminal and instead utilizing the benefits of developing within a notebook. This section explains how the Spark integration within a Jupyter notebook takes place.

1. We will create a command function, sparknotebook, that we can call from the terminal to open up a Spark session through Jupyter notebooks from the Anaconda installation. This requires two settings to be set in the .bashrc file:
   1. PySpark Python be set to python 3
   2. PySpark driver for python to be set to Jupyter
2. The sparknotebook function can now be accessed directly from the terminal by executing the following command:

$ sparknotebook

3. The function should then initiate a brand new Jupyter notebook session through the default web browser. A new Python script within Jupyter notebooks with a .ipynb extension can be created by clicking on the New button on the right-hand side and by selecting Python 3 under Notebook: as seen in the following screenshot:

4. Once again, just as was done at the terminal level for Spark, a simple script of sc will be executed within the notebook to confirm that Spark is up and running through Jupyter:

5. Ideally, the Version, Master, and AppName should be identical to the earlier output when sc was executed at the terminal. If this is the case, then PySpark has been successfully installed and configured to work with Jupyter notebooks.

It is important to note that if we were to call a Jupyter notebook through the terminal without specifying sparknotebook, our Spark session will never be initiated and we will receive an error when executing the SparkContext script.

We can access a traditional Jupyter notebook by executing the following at the terminal:

jupyter-notebook

Once we start the notebook, we can try and execute the same script for sc.master as we did previously, but this time we will receive the following error:

There are many managed offerings online of companies offering Spark through a notebook interface where the installation and configuration of Spark with a notebook have already been managed for you. These are the following:

• Hortonworks (https://hortonworks.com/)
• Cloudera (https://www.cloudera.com/)
• MapR (https://mapr.com/)
• DataBricks (https://databricks.com/)

For most chapters, one of the first things that we will do is to initialize and configure our Spark cluster.

Import the following before initializing cluster.

• from pyspark.sql import SparkSession

This section walks through the steps to initialize and configure a Spark cluster.

1. Import SparkSession using the following script:

from pyspark.sql import SparkSession

2. Configure SparkSession with a variable named spark using the following script:

spark = SparkSession.builder \
    .master("local[*]") \
    .appName("GenericAppName") \
    .config("spark.executor.memory", "6gb") \
.getOrCreate()

This section explains how the SparkSession works as an entry point to develop within Spark.

1. Staring with Spark 2.0, it is no longer necessary to create a SparkConf and SparkContext to begin development in Spark. Those steps are no longer needed as importing SparkSession will handle initializing a cluster.  Additionally, it is important to note that SparkSession is part of the sql module from pyspark.
2. We can assign properties to our SparkSession:
   1. master: assigns the Spark master URL to run on our local machine with the maximum available number of cores.  
   2. appName: assign a name for the application
   3.  config: assign 6gb to the spark.executor.memory
   4. getOrCreate: ensures that a SparkSession is created if one is not available and retrieves an existing one if it is available

For development purposes, while we are building an application on smaller datasets, we can just use master("local").  If we were to deploy on a production environment, we would want to specify master("local[*]") to ensure we are using the maximum cores available and get optimal performance.

To learn more about SparkSession.builder, visit the following website:

https://spark.apache.org/docs/2.2.0/api/java/org/apache/spark/sql/SparkSession.Builder.html

Once we are done developing on our cluster, it is ideal to shut it down and preserve resources.

This section walks through the steps to stop the SparkSession.

1. Execute the following script:

spark.stop()

2. Confirm that the session has closed by executing the following script:

sc.master

This section explains how to confirm that a Spark cluster has been shut down.

1. If the cluster has been shut down, you will receive the error message seen in the following screenshot when executing another Spark command in the notebook:

Shutting down Spark clusters may not be as critical when working in a local environment; however, it will prove costly when Spark is deployed in a cloud environment where you are charged for compute power.