Applied Supervised Learning with Python: Use scikit-learn to build predictive models from real-world datasets and prepare yourself for the future of machine learning

Say we have a problem statement that involves predicting whether a particular earthquake caused a tsunami or not. How do we decide what model to use? What do we know about the data we have? Nothing! But if we don't know and understand our data, chances are we'll end up building a model that's not very interpretable or reliable.

When it comes to data science, it's important to have a thorough understanding of the data we're dealing with, in order to generate features that are highly informative and, consequently, to build accurate and powerful models.

In order to gain this understanding, we perform an exploratory analysis on the data to see what the data can tell us about the relationships between the features and the target variable. Getting to know our data will even help us interpret the model we build and identify ways we can improve its accuracy.

The approach we take to achieve this is to allow the data to reveal its structure or model, which helps gain some new, often unsuspected...

In order to find out what our data really looks like, we use a technique known as data profiling. This is defined as the process of examining the data available from an existing information source (for example, a database or a file) and collecting statistics or informative summaries about that data. The goal is to make sure that you understand your data well and are able to identify any challenges that the data may pose early on in the project, which is done by summarizing the dataset and assessing its structure, content, and quality.

Data profiling includes collecting descriptive statistics and data types. Here are a few commands that are commonly used to get a summary of a dataset:

When there is no value (that is, a null value) recorded for a particular feature in a data point, we say the data is missing. Having missing values in a real dataset is inevitable; no dataset is ever perfect. However, it is important to understand why the data is missing, and if there is a factor that has affected the loss of data. Appreciating and recognizing this allows us to handle the remaining data in an appropriate manner. For example, if the data is missing randomly, then it's highly likely that the remaining data is still representative of the population. However, if the missing data is not random in nature and we assume that it is, it could bias our analysis and subsequent modeling.

Let's look at the common reasons (or mechanisms) for missing data:

In this section, we'll look at how individual variables behave—what kind of values they take, what the distribution across those values is, and how those distributions can be represented visually.

There are two reasons why it is important to find relationships between variables in the data:

Now, there are a number of ways we can visualize these relationships, and this really depends on the types of variable we are trying to find the relationship between, and how many we are considering as part of the equation or comparison.

In this chapter, we started by talking about why data exploration is an important part of the modeling process and how it can help in not only preprocessing the dataset for the modeling process, but also help us engineer informative features and improve model accuracy. This chapter focused on not only gaining a basic overview of the dataset and its features, but also gaining insights by creating visualizations that combine several features.

We looked at how to find the summary statistics of a dataset using core functionality from pandas. We looked at how to find missing values and talked about why they're important, while learning how to use the Missingno library to analyze them and the pandas and scikit-learn libraries to impute the missing values.

Then, we looked at how to study the univariate distributions of variables in the dataset and visualize them for both categorical and continuous variables using bar charts, pie charts, and histograms. Lastly, we learned how to explore relationships...