Introduction

Data cleaning and preparation takes about 70% of the effort in the entire process of a machine learning project. This step is essential because the quality of the data determines the accuracy of the prediction model. A clean dataset should contain good samples of the scenarios that we want to predict, and this will give us good prediction results. Also, the data should be balanced, which means that every category we want to predict should have similar number of samples. For example, if we want to predict whether it will rain or not on any particular day, and if the sample data size is 100, the data could contain 40 samples for It will rain and 60 samples for It will not rain today, or vice versa. However, if the ratio is 20:80 or 30:70, it is an unbalanced dataset, and this will not yield good results for the minority class.

In the following section, we will look at the essential operations performed on data frames in R. These operations will help us to manipulate and analyze...

Advanced Operations on Data Frames

In the previous chapter, we performed a number of operations on data frames, including rbind(). There are many more operations that can be performed on data frames, which are very useful while preparing the data for the model. The following exercises will describe these operations in detail and illustrate them through their corresponding implementation in R:

• The order function: The order function is used to sort a data frame. We can specify ascending or descending order using the "-" symbol.
• The sort function: The sort function can also be used to sort the data. The order can be specified as "decreasing=TRUE" or "decreasing"="FALSE".
• The rank function: The rank function is used to rank the values in the data in a numerical manner.

Sorting, ordering, and ranking are operations that act as techniques to identify outliers. Outliers are values that are either too big or too small and do not fit in the value...

Identifying the Input and Output Variables

For any dataset, we should identify the input variables and the output variables. For the iris dataset, the input variables are the following:

1. SepalLength
2. SepalWidth
3. PetalLength
4. PetalWidth

The output variable, or the field to be predicted, is Species.

Identifying the Category of Prediction

Based on the category of prediction, we will perform different pre-processing steps. The category of prediction could be any of these:

• Categorical Prediction: In this type of prediction, the output to be predicted will have class values such as yes, no, or given categories.
• Numeric Prediction: In a numeric prediction, the output that will be predicted is a numeric value, such as predicting the cost of a house.

Handling Missing Values, Duplicates, and Outliers

In any dataset, we might have missing values, duplicate values, or outliers. We need to ensure that these are handled appropriately so that the data used by the model is clean.

Handling Missing Values

Missing values in a data frame can affect the model during the training process. Therefore, they need to be identified and handled during the pre-processing stage. They are represented as NA in a data frame. Using the example that follows, we will see how to identify a missing value in a dataset.

Using the is.na(), complete.cases(), and md.pattern() functions, we will identify the missing values.

The is.na() function, as the name suggests, returns TRUE for those elements marked NA or, for numeric or complex vectors, NaN (Not a Number) , and FALSE. The complete.cases() function returns TRUE if the value is missing and md.pattern() gives a summary of the missing values.

Exercise 12: Identifying the Missing Values

In the following example, we are adding...

Handling Outliers

Any datapoint with a value that is very different from the other data points is an outlier. Outliers can affect the training process negatively and therefore they need to be handled gracefully. In the following section, we will illustrate via examples both the process of detecting an outlier and the techniques used to handle them.

Exercise 16: Identifying Outlier Values

The outlier package can detect the outlier values. Using the opposite=TRUE parameter will fetch the outliers from the other side of dataset. The outlier values can be verified using a boxplot.

1. Attach the outlier package:

   library(outliers)

2. Detect outliers:

   #Detect outliers

   outlier(PimaIndiansDiabetes[,1:4])

   The output is as follows:

   pregnant  glucose pressure  triceps

         17        0        0       99

   Detect outliers from the other end:

   #This...

Summary

In this chapter, we learned how to perform several operations on a data frame, including scaling, standardizing, and normalizing. Also, we covered the sorting, ranking, and joining operations with their implementations in R. We discussed the need for pre-processing of the data; and identified and handled outliers, missing values, and duplicate values.

Next, we moved on to the sampling of data. It is important for the data to contain a reasonable sample of each class that is to be predicted. If the data is imbalanced, it can affect our predictions in a negative manner. Therefore, we can use either the undersampling, oversampling, ROSE, or SMOTE techniques imbalanced to ensure that the dataset is representative of all the classes that we want to predict. This can be done using the MICE, rpart, ROSE, and caret packages.

In the next chapter, we will cover feature engineering in detail, where we will focus on extracting features to create models.