Converting a PySpark dataframe to an array

In order to form the building blocks of the neural network, the PySpark dataframe must be converted into an array. Python has a very powerful library, numpy, that makes working with arrays simple.

Getting ready

The numpy library should be already available with the installation of the anaconda3 Python package. However, if for some reason the numpy library is not available, it can be installed using the following command at the terminal:

pip install or sudo pip install will confirm whether the requirements are already satisfied by using the requested library:

import numpy as np

...