Chapter 2. Exploratory Analysis of Data

Note

Learning Objectives

By the end of this chapter, you will be able to:

Define a problem statement with industry standard frameworks
Perform univariate and bivariate analysis
Explain multivariate analysis
Perform hypothesis testing
Perform data wrangling using the dplyr and reshape2 packages
Visualize data using the ggplot2 package

Note

In this chapter, we will acquaint learners with techniques to clean, transform, and visualize data in order to get useful insights.

Introduction

Chapter 1, R for Advanced Analytics, introduced to you the R language and its ecosystem for data science. We are now ready to enter a crucial part of data science and machine learning, that is, Exploratory Data Analysis (EDA), the art of understanding the data.

In this chapter, we will approach EDA with the same banking dataset used in the previous chapter, but in a more problem-centric way. We will start by defining the problem statement with industry standard artifacts, design a solution for the problem, and learn how EDA fits in the larger problem framework. We will then tackle the EDA for the direct marketing campaigns (phone calls) of a Portuguese banking institution use case using a combination of data engineering, data wrangling, and data visualization techniques in R, backed up by a business-centric approach.

In any data science use case, understanding the data consumes the bulk of the time and effort. Most data science professionals spend around 80% of their time understanding...

Defining the Problem Statement

If you recollect the data we explored in Chapter 1, R for Advanced Analytics, bank marketing data, we have a dataset that captures the telemarketing campaigns conducted by a bank to attract customers.

A large multinational bank is designing a marketing campaign to achieve its growth target by enticing customers for bank deposits. The campaign has been ineffective in luring customers, and the marketing team wants to understand how the campaign can be improved to achieve the growth targets.

We can reframe the problem from the business stakeholders' perspective and try to see what kind of solution would best fit here.

Problem-Designing Artifacts

Just like there are several frameworks, templates, and artifacts for software engineering and other industrial projects, data science and business analytics projects can also be effectively represented using industry standard artifacts. Some popular choices are available from consulting giants such as McKinsey, BCG, and decision...

Understanding the Science Behind EDA

In layman's terms, we can define EDA as the science of understanding data. A more formal definition is the process of analyzing and exploring datasets to summarize its characteristics, properties, and latent relationships using statistical, visual, analytical, or a combination of techniques.

To cement our understanding, let's break down the definition further. The dataset is a combination of numeric and categorical features. To study the data, we might need to explore features individually, and to study relationships, we might need to explore features together. Depending on the number of features and the type of features, we may cross paths with different types of EDA.

To simplify, we can broadly classify the process of EDA as follows:

Univariate analysis: Studying a single feature
Bivariate analysis: Studying the relationship between two features
Multivariate analysis: Studying the relationship between more than two features

For now, we will restrict the scope...

Exploratory Data Analysis

We will get started with the dataset available to download from UCI ML Repository at https://archive.ics.uci.edu/ml/datasets/Bank%20Marketing.

Download the ZIP file and extract it to a folder in your workspace and use the file named bank-additional-full.csv. Ask the students to start a new Jupyter notebook or an IDE of their choice and load the data into memory.

Exercise 18: Studying the Data Dimensions

Let's quickly ingest the data using the simple commands we explored in the previous chapter and take a look at a few essential characteristics of the dataset.

We are exploring the length and breadth of the data, that is, the number of rows and columns, the names of each column, the data type of each column, and a high-level view of what is stored in each column.

Perform the following steps to explore the bank dataset:

First, import all the required libraries in RStudio:

library(dplyr)
library(ggplot2)
library(repr)
library(cowplot)

Now, use the option method to set the width...

Univariate Analysis

Univariate analysis is the study of a single feature/variable. Here, we describe the data to help us get an overall view of how it is organized. For numeric features, such as age, duration, nr.employed (numeric features in the dataset) and many others, we look at summary statistics such as min, max, mean, standard deviation, and percentile distribution. These measures together help us understand the spread of the data. Similarly, for categorical features such as job, marital, and education, we need to study the distinct values in the feature and the frequency of these values. To accomplish this, we can implement a few analytical, visual, and statistical techniques. Let's take a look at the analytical and visual techniques for exploring numeric features.

Exploring Numeric/Continuous Features

If you explored the previous output snippet, you might have noted that we have a mix of numeric and categorical features in the dataset. Let's start by looking at the first feature in...

Exploring Categorical Features

Categorical features differ from numeric or continuous features in nature, and therefore the traditional methods used earlier aren't applicable here. We can analyze the number of different classes within a categorical variable and the frequency associated with each. This can be achieved using either simple analytical techniques or visual techniques. Let's explore a list of categorical features using a combination of both.

Exercise 24: Exploring Categorical Features

In this exercise, we will start with a simple variable, that is, marital, which indicates the marital status of the client. Let's use the dplyr library to perform grouped data aggregation.

Perform the following steps to complete the exercise:

First, import the dplyr library in the system using the following command:
```
library(dplyr)
```
Next, we will create an object named marital_distribution and store the value based on the following condition:
```
marital_distribution <- df %>% group_by(marital) %>% ...
```

Bivariate Analysis

In bivariate analysis, we extend our analysis to study two variables together. In our use case, we have around 20 independent variables. It is indeed possible to study all permutation combinations of the available 20 variables, but we won't go to that extent in this chapter. In our use case, we are more interested in studying all the factors that led to the poor performance of the campaign. Therefore, our primary focus will be to perform bivariate analysis and study the relationship between all the independent variables and our dependent target variable. Again, depending on the type of variable, we will have a different type of visual or analytical technique to analyze the relationship between the two variables. The possible combinations are numeric and numeric, and numeric and categorical. Given that our dependent variable is a categorical variable, we might have to explore the relationship between two independent variables in our list to study the relationship between...

Studying the Relationship between Two Numeric Variables

To understand how we can study the relationship between two numeric variables, we can leverage scatter plots. It is a 2-dimensional visualization of the data, where each variable is plotted on an axis along its length. Relationships between the variables are easily identified by studying the trend across the visualization. Let's take a look at an example in the following exercise.

Exercise 30: Studying the Relationship between Employee Variance Rate and Number of Employees

Let's study the relationship between employee variance rate and the number of employees. Ideally, the number of employees should increase as the variation rate increases.

Perform the following steps to complete the exercise:

First, import the ggplot2 package using the following command:
```
library(ggplot2)
```
Create a DataFrame object, df, and use the bank-additional-full.csv file using the following command:
```
df <- read.csv("/Chapter 2/Data/bank-additional/bank-additional-full...
```

Studying the Relationship between a Categorical and a Numeric Variable

Let's first recall the methods discussed to study the relationship between the numeric and categorical variable and discuss the approach to execute it.

In this section, we will discuss the different aggregation metrics that we can use for summarizing the data. So far, we have used avg, but a better approach would be to use a combination of avg, min, max, and other metrics.

Exercise 31: Studying the Relationship between the y and age Variables

We have a categorical dependent variable and nine numeric variables to explore. To start small, we will first explore the relationship between our target, y, and age. To study the relationship between a categorical and numeric variable, we can choose a simple analytical technique where we calculate the average age across each target outcome; if we see stark differences, we can make insights from the observations.

In this exercise, we will calculate the average age across each target...

Studying the Relationship Between Two Categorical Variables

To study the relationship and patterns that exist between two categorical variables, we can first explore the frequency distribution across each category of the variables. A higher concentration in any outcome might be a potential insight. The most effective way to visualize this is using stacked bar charts.

A stacked bar chart will help us to observe the distribution of the target variable across multiple categorical variables. The distribution will reveal whether a specific category in a categorical variable dominates the target variable, y. If yes, we can further explore its influence on our problem.

In the next few exercises, we will explore various categorical variables across target variable y using stacked bar chart. We will plot absolute count and percentage to understand the distribution better.

Exercise 34: Studying the Relationship Between the Target y and marital status Variables

In this exercise, we will demonstrate the...

Multivariate Analysis

Multivariate analysis is the process of studying the relationships between more than two variables; essentially, one dependent variable and more than one independent variable. Bivariate analysis is a form of multivariate analysis. There are several forms of multivariate analysis that are important, but we will skip the details for now to restrict the scope of the chapter. In the next few chapters, we will take a closer look at linear and logistic regression, which are two popular multivariate analysis techniques.

Some of the most common techniques used in multivariate analysis are as follows:

Multiple linear regression (studying the impact of more than one independent variable on a numeric/continuous target variable)
Logistic regression (studying the impact of more than one independent variable on a categorical target variable)
Factor analysis
MANOVA

Validating Insights Using Statistical Tests

Throughout the journey of EDA, we have collected and noted some interesting patterns for further validation. It is now the right time to test whether whatever we observed previously are actually valid patterns or just appeared to be interesting due to random chance. The most effective and straightforward way to approach this validation is by performing a set of statistical tests and measuring the statistical significance of the pattern. We have a ton of options in the available set of tests to choose from. The options vary based on the type of independent and dependent variable. The following is a handy reference diagram that explains the types of statistical test that we can perform to validate our observed patterns:

Figure 2.24: Validating dependent and independent variables

Let's collect all our interesting patterns into one place here:

The campaign outcome has a higher chance of yes when the employee variance rate is low.
The campaign outcome has...

Categorical Dependent and Numeric/Continuous Independent Variables

Hypotheses 1 and 2 have a continuous independent variable. Referring to the figure in the previous section, we will opt for the chi-squared test. In the process of hypothesis testing, we start by defining a null hypothesis and an alternate hypothesis. Start with a negative approach, that is, assume the null hypothesis to be what we don't want to happen. The hypothesis test examines the chances that the pattern observed happens due to random chance or there if is certainty about the observation. This measure is quantified as probability. If the probability of the significance of the null hypothesis to happen is less than 5% (or a suitable cut-off), we reject the null hypothesis and confirm the validity of the alternate hypothesis.

Let's begin; for hypothesis 1, we define the following:

Null hypothesis: The campaign outcome has no relationship with the employee variance rate.
Alternate hypothesis: The campaign outcome has a relationship...

Categorical Dependent and Categorical Independent Variables

Moving on, let's take a look at the third hypothesis. To test the relationship between the categorical dependent variable and categorical independent variable, we can use the chi squared test.

For hypothesis 3, we define the following:

Null hypothesis: The campaign outcome has no relationship with clients who never married.
Alternate hypothesis: The campaign outcome has a relationship with clients who never married.

In the following exercise, we will leverage R's chi-square test function to validate the hypothesis..

Exercise 38: Hypothesis 3 Testing for Categorical Dependent Variables and Categorical Independent Variables

In this exercise, we will perform a statistical test using the chi-squared test. We use the chi-squared test because both the independent and dependent variables are categorical, particularly when testing the relationship between y and marital status.

Perform the following steps:

Import the required libraries and create...

Summary

In this chapter, we explored EDA using a practical use case and traversed the business problem. We started by understanding the overall process of executing a data science problem and then defined our business problem using an industry standard framework. With the use case being cemented with appropriate questions and complications, we understood the role of EDA in designing the solution for the problem. Exploring the journey of EDA, we studied univariate, bivariate, and multivariate analysis. We performed the analysis using a combination of analytical as well as visual techniques. Through this, we explored the R packages for visualization, that is, ggplot and some packages for data wrangling through dplyr. We also validated our insights with statistical tests and, finally, collated the insights noted to loop back with the original problem statement.

In the next chapter, we will lay the foundation for various machine learning algorithms, and discuss supervised learning in depth.