iPython

Finally, let's fire up iPython and interact with the SparkContext object. As mentioned in Chapter 3, Building and Running a Spark Application, refer to the iPython site (http://jupyter.readthedocs.org/en/latest/install.html) for installing the Jupyter and iPython system.

First, change the directory to fdps-v3, where you would have downloaded the code and data for this book:

cd ~/fdps-v3

The command to start iPython is as follows:

PYSPARK_DRIVER_PYTHON=ipython PYSPARK_DRIVER_PYTHON_OPTS="notebook" ~/Downloads/spark-2.0.0/bin/pyspark

The iPython notebook will be launched in the web browser, as shown in the following screenshot, and you will see a list of iPython notebooks:

Click on the 000-PreFlightCheck.ipynb notebook:

Run the first cell using Shift + Enter. You will see the results, including the Python version, Spark version, and so on, as shown in the preceding screenshot. The notebook has more cells, which we will see in the next few chapters.

Now that you are able to create a...