Writing and running a Spark application from Jupyter Notebook

Before you run the following steps, make sure that you grasped the components and their interactions that we have introduced in the previous section of this chapter:

1. Validate that the Spark operator pod is running by running the following command:

   kubectl get pods -n ml-workshop | grep spark-operator

You should see the following response:

Figure 5.35 – Spark operator pod

2. Validate that the JupyterHub pod is running by running the following command:

   kubectl get pods -n ml-workshop | grep jupyterhub

You should see the following response:

Figure 5.36 – JupyterHub pod

3. Before you start the notebook, let's delete the Spark cluster you have created in the previous sections by running the following command. This is to demonstrate that JupyterHub will automatically create a new instance of Spark cluster for you:

   kubectl delete sparkcluster...