Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Conferences
Free Learning
Arrow right icon
Arrow up icon
GO TO TOP
Applied Supervised Learning with Python

You're reading from   Applied Supervised Learning with Python Use scikit-learn to build predictive models from real-world datasets and prepare yourself for the future of machine learning

Arrow left icon
Product type Paperback
Published in Apr 2019
Publisher
ISBN-13 9781789954920
Length 404 pages
Edition 1st Edition
Languages
Arrow right icon
Authors (2):
Arrow left icon
Ishita Mathur Ishita Mathur
Author Profile Icon Ishita Mathur
Ishita Mathur
Benjamin Johnston Benjamin Johnston
Author Profile Icon Benjamin Johnston
Benjamin Johnston
Arrow right icon
View More author details
Toc

Chapter 2: Exploratory Data Analysis and Visualization


Activity 2: Summary Statistics and Missing Values

Solution

The steps to complete this activity are as follows:

  1. Read the data. Use pandas' .read_csv method to read the CSV file into a pandas DataFrame:

    data = pd.read_csv('house_prices.csv')
  2. Use pandas' .info() and .describe() methods to view the summary statistics of the dataset:

    data.info()
    data.describe().T

    The output of info() will be:

    Figure 2.39: The output of the info() method

    The output of describe() will be:

    Figure 2.40: The output of the describe() method

  3. Find the total count and total percentage of missing values in each column of the DataFrame and display them for columns having at least one null value, in descending order of missing percentages.

    As we did in Exercise 12: Visualizing Missing Values, we will use the .isnull() function on the DataFrame to get a mask, find the count of null values in each column by using the .sum() function over the mask DataFrame and the fraction of null values by using .mean() over the mask DataFrame and multiply by 100 to convert it to a percentage. Then, we use pd.concat() to combine the total and percentage of null values into a single DataFrame and sort the rows by percentage of missing values:

    mask = data.isnull()
    total = mask.sum()
    percent = 100*mask.mean()
    
    missing_data = pd.concat([total, percent], axis=1,join='outer',
                   keys=['count_missing', 'perc_missing'])
    missing_data.sort_values(by='perc_missing', ascending=False, inplace=True)
    
    missing_data[missing_data.count_missing > 0]

    The output will be:

    Figure 2.41: Total count and percentage of missing values in each column

  4. Plot the nullity matrix and nullity correlation heatmap. First, we find the list of column names for those having at least one null value. Then, we use the missingno library to plot the nullity matrix (as we did in Exercise 12: Visualizing Missing Values) for a sample of 500 points, and the nullity correlation heatmap for the data in those columns:

    nullable_columns = data.columns[mask.any()].tolist()
    msno.matrix(data[nullable_columns].sample(500))
    plt.show()
    
    msno.heatmap(data[nullable_columns], figsize=(18,18))
    plt.show()

    The nullity matrix will look like this:

    Figure 2.42: Nullity matrix

    The nullity correlation heatmap will look like this:

    Figure 2.43: Nullity correlation heatmap

  5. Delete the columns having more than 80% of values missing. Use the .loc operator on the DataFrame we created in Step 3 to select only those columns that had less than 80% of values missing:

    data = data.loc[:,missing_data[missing_data.perc_missing < 80].index]
  6. Replace null values in the FireplaceQu column with NA values. Use the .fillna() method to replace null values with the NA string:

    data['FireplaceQu'] = data['FireplaceQu'].fillna('NA')

Activity 3: Visually Representing the Distribution of Values

Solution

  1. Plot a histogram using Matplotlib for the target variable, SalePrice. First, we initialize the figure using the plt.figure command and set the figure size. Then, we use Matplotlib's .hist() function as our primary plotting function, to which we pass the SalePrice series object for plotting the histogram. Lastly, we specify the axes labels and show the plot:

    plt.figure(figsize=(8,6))
    
    plt.hist(data.SalePrice, bins=range(0,800000,50000))
    
    plt.ylabel('Number of data points')
    plt.xlabel('SalePrice')
    plt.show()

    The output will be as follows:

    Figure 2.44: Histogram for the target variable

  2. Find the number of unique values within each column having the object type. Create a new DataFrame called object_variables by using the .select_dtypes function on the original DataFrame to select those columns with the numpy.object data type. Then, find the number of unique values for each column in this DataFrame by using the .nunique() function, and sort the resultant series:

    object_variables = data.select_dtypes(include=[np.object])
    object_variables.nunique().sort_values()

    The output will be:

    Figure 2.45: Number of unique values within each column having the object type

  3. Create a DataFrame representing the number of occurrences for each categorical value in the HouseStyle column. Use the .value_counts() function to calculate the frequencies of each value in decreasing order in the form of a pandas series, then reset the index to give us a DataFrame and sort the values by the index:

    counts = data.HouseStyle.value_counts(dropna=False)
    counts.reset_index().sort_values(by='index')

    The output will be:

    Figure 2.46: Number of occurrences for each categorical value in the HouseStyle column

  4. Plot a pie chart representing these counts. As in Step 1, we initialize the image using plt.figure() and use the plt.title() and plt.show() methods to set the figure title and display it respectively. The primary plotting function used is plt.pie(), to which we pass the series we created in the previous step:

    plt.figure(figsize=(10,10))
    plt.pie(counts, labels=counts.index)
    plt.title('Pie chart showing counts for\nHouseStyle categories')
    plt.show()

    The output will be as follows:

    Figure 2.47: Pie chart representing the counts

  5. Find the number of unique values within each column having the number type. As done in Step 2, now select columns having the numpy.number data type and find the number of unique values in each column using .nunique(). Sort the resultant series in descending order:

    numeric_variables = data.select_dtypes(include=[np.number])
    numeric_variables.nunique().sort_values(ascending=False)

    The output will be as follows:

    Figure 2.48: Number of unique values within each column having the number type

  6. Plot a histogram using Seaborn for the LotArea variable. Use Seaborn's .distplot() function as the primary plotting function, to which the LotArea series in the DataFrame needs to be passed (without any null values; use .dropna() on the series to remove them). To improve the plot view, also set the bins parameter and specify the X axis limits using plt.xlim():

    plt.figure(figsize=(10,7))
    sns.distplot(data.LotArea.dropna(), , bins=range(0,100000,1000))
    plt.xlim(0,100000)
    plt.show()

    The output will be as follows:

    Figure 2.49: Histogram for the LotArea variable

  7. Calculate the skew and kurtosis values for the values in each column:

    data.skew().sort_values()
    data.kurt()

    The output for skew values will be:

    Figure 2.50: Skew values for each column

    The output for kurtosis values will be:

    Figure 2.51: Kurtosis values for each column

Activity 4: Relationships Within the Data

Solution

  1. Plot the correlation heatmap for the dataset. As we did in Exercise 23: Correlation Heatmap, plot the heatmap using Seaborn's .heatmap() function and pass the feature correlation matrix (as determined by using pandas' .corr() function on the DataFrame). Additionally, set the color map to RdBu using the cmap parameter and the minimum and maximum values on the color scale to -1 and 1 using the vmin and vmax parameters respectively:

    plt.figure(figsize = (12,10))
    sns.heatmap(data.corr(), square=True, cmap="RdBu", vmin=-1, vmax=1)
    plt.show()

    The output will be:

    Figure 2.52: Heatmap for the dataset

  2. Plot a more compact heatmap having annotations for correlation values using the following subset of features:

    feature_subset = [
        'GarageArea', 'GarageCars','GarageCond','GarageFinish','GarageQual','GarageType',
        'GarageYrBlt','GrLivArea','LotArea','MasVnrArea','SalePrice'
    ]

    Now do the same as in the previous step, this time selecting only the above columns in the dataset, and adding an annot parameter with the True value to the primary plotting function, all else remaining the same:

    plt.figure(figsize = (12,10))
    sns.heatmap(data[feature_subset].corr(), square=True, annot=True, cmap="RdBu", vmin=-1, vmax=1)
    plt.show()

    The output will be as follows:

    Figure 2.53: Heatmap with annotations for correlation values

  3. Display the pairplot for the same subset of features, with the KDE plot on the diagonals and scatter plot elsewhere. Use Seaborn's .pairplot() function to plot the pairplot for the non-null values in the selected columns of the DataFrame. To make the diagonal plots KDE plots, pass kde to the diag_kind parameter and to set all other plots as scatter plots, pass scatter to the kind parameter:

    sns.pairplot(data[feature_subset].dropna(), kind ='scatter', diag_kind='kde')
    plt.show()

    The output will be:

    Figure 2.54: Pairplot for the same subset of features

  4. Create a boxplot to show the variation in SalePrice for each category of GarageCars. The primary plotting function used here will be Seaborn's .boxplot() function, to which we pass the DataFrame along with parameters x and y, the former is the categorical variable and the latter is the continuous variable over which we want to see the variation within each category, that is, GarageCars and SalePrice respectively:

    plt.figure(figsize=(10, 10))
    sns.boxplot(x='GarageCars', y="SalePrice", data=data)
    plt.show()

    The output will be as follows:

    Figure 2.55: Boxplot showing variation in SalePrice for each category of GarageCars

  5. Plot a line graph using Seaborn to show the variation in SalePrice for older and more recently built flats. Here, we will plot a line plot using Seaborn's .lineplot() function. Since we want to see the variation in SalePrice, we take this as the y variable, and as the variation is across a period of time, we take YearBuilt as the x variable. Keeping this in mind, we pass the respective series as values to the y and x parameters for the primary plotting function. We also pass a ci=None parameter to hide the standard deviation indicator around the line in the plot:

    plt.figure(figsize=(10,7))
    sns.lineplot(x=data.YearBuilt, y=data.SalePrice, ci=None)
    plt.show()

    The output will be as follows:

    Figure 2.56: Line graph showing the variation in SalePrice for older and more recently built flats

lock icon The rest of the chapter is locked
Register for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Renews at $19.99/month. Cancel anytime
Banner background image