Data Cleansing

In machine learning, one of the first tasks that you need to perform is data cleansing. Very seldom would you have a dataset that you can use straightaway to train your model. Instead, you have to examine the data carefully for any missing values and either remove them or replace them with some valid values, or you have to normalize them if there are columns with wildly different values. The following sections show some of the common tasks you need to perform when cleaning

Cleaning Rows with NaNs

Consider a CSV file named NaNDataset.csv with the following content:

A,B,C
1,2,3
4,,6
7,,9
10,11,12
13,14,15
16,17,18 

Visually, you can spot that there are a few rows with empty fields. Specifically, the second and third rows have missing values for the second columns. For small sets of data, this is easy to spot. But if you have a large dataset, it becomes almost impossible to detect. An effective way to detect for empty rows is to load the dataset into a Pandas dataframe and...