Loading data on to Spark RDDs

In this section, we are going to look at loading data on to Spark RDDs, and will cover the following topics:

• The UCI machine learning data repository
• Getting data from the repository to Python
• Getting data into Spark

Let's start with an overview of the UCI machine learning data repository.

The UCI machine learning repository

We can access the UCI machine learning repository by navigating to https://archive.ics.uci.edu/ml/. So, what is the UCI machine learning repository? UCI stands for the University of California Irvine machine learning repository, and it is a very useful resource for getting open source and free datasets for machine learning. Although PySpark's main issue or solution doesn't concern machine learning, we can use this as a chance to get big datasets that help us test out the functions of PySpark.

Let's take a look at the KDD Cup 1999 dataset, which we will download, and then we will load the whole dataset into PySpark.

Getting the data from the repository to Spark

We can follow these steps to download the dataset and load it in PySpark:

1. Click on Data Folder.
2. You will be redirected to a folder that has various files as follows:

You can see that there's kddcup.data.gz, and there is also 10% of that data available in kddcup.data_10_percent.gz. We will be working with food datasets. To work with the food datasets, right-click on kddcup.data.gz, select Copy link address, and then go back to the PySpark console and import the data.

Let's take a look at how this works using the following steps:

1. After launching PySpark, the first thing we need to do is import urllib, which is a library that allows us to interact with resources on the internet, as follows:

import urllib.request

2. The next thing to do is use this request library to pull some resources from the internet, as shown in the following code:

f = urllib.request.urlretrieve("https://archive.ics.uci.edu/ml/machine-learning-databases/kddcup99-mld/kddcup.data.gz"),"kddcup.data.gz"

This command will take some time to process. Once the file has been downloaded, we can see that Python has returned and the console is active.

3. Next, load this using SparkContext. So, SparkContext is materialized or objectified in Python as the sc variable, as follows:

sc

This output is as demonstrated in the following code snippet:

SparkContext
Spark UI
Version
 v2.3.3
Master
 local[*]
AppName
 PySparkShell

Getting data into Spark

1. Next, load the KDD cup data into PySpark using sc, as shown in the following command:

raw_data = sc.textFile("./kddcup.data.gz")

2. In the following command, we can see that the raw data is now in the raw_data variable:

raw_data

This output is as demonstrated in the following code snippet:

./kddcup.data,gz MapPartitionsRDD[3] at textFile at NativeMethodAccessorImpl.java:0

If we enter the raw_data variable, it gives us details regarding kddcup.data.gz, where raw data underlying the data file is located, and tells us about MapPartitionsRDD.

Now that we know how to load the data into Spark, let's learn about parallelization with Spark RDDs.