When given a new dataset, it is first important to recognize whether or not your data is structured or unstructured:

• Structured (organized) data: Data that can be broken down into observations and characteristics. They are generally organized using a tabular method (where rows are observations and columns are characteristics).

• Unstructured (unorganized) data: Data that exists as a free-flowing entity and does not follow standard organizational hierarchy such as tabularity. Often, unstructured data appears to us as a blob of data, or as a single characteristic (column).

A few examples that highlight the difference between structured and unstructured data are as follows:

• Data that exists in a raw free-text form, including server logs and tweets, are unstructured

• Meteorological data, as reported by scientific instruments...

# Import our data manipulation tool, Pandas
import pandas as pd
# Create a pandas DataFrame from some unstructured Server Logs
logs = pd.read_table('../data/server_logs.txt', header=None, names=['Info'])

# header=None, specifies that the first line of data is the first data point, not a column name
# names=['Info] is me setting the column name in our DataFrame for easier access

# Look at the first 5...

To accomplish our diagnoses of the various types of data, we will begin with the highest order of separation. When dealing with structured, tabular data (which we usually will be doing), the first question we generally ask ourselves is whether the values are of a numeric or categorical nature.

Quantitative data are data that are numerical in nature. They should be measuring the quantity of something.

Qualitative data are data that are categorical in nature. They should be describing the quality of something.

Basic examples:

• Weather measured as temperature in Fahrenheit or Celsius would be quantitative
• Weather measured as cloudy or sunny would be qualitative
• The name of a person visiting the White House would be qualitative
• The amount of blood you donate at a blood drive is quantitative

The first two examples show that we can describe...

We already know that we can identify data as being either qualitative or quantitative. But, from there, we can go further. The four levels of data are:

• The nominal level
• The ordinal level
• The interval level
• The ratio level

Each level comes with a varying level of control and mathematical possibilities. It is crucial to know which level data lives on because it will dictate the types of visualizations and operations you are allowed to perform.

The first level of data, the nominal level, has the weakest structure. It consists of data that are purely described by name. Basic examples include blood type (A, O, AB), species of animal, or names of people. These types of data are all qualitative...

Understanding the various levels of data is necessary to perform feature engineering. When it comes time to build new features, or fix old ones, we must have ways of identifying how to work with every column.

Here is a quick table to summarize what is and isn't possible at every level:

Understanding the features that we are working with is step zero of feature engineering. If we cannot understand the data given to us, we will never hope to fix, create, and utilize features in order to create well-performing, machine-learning pipelines. In this chapter, we were able to recognize, and extract the levels of data from our datasets and use that information to create useful and meaningful visuals that shine new lights on our data.

In the next chapter, we will use all of this new-found knowledge of the levels of data to start improving our features, and we will start to use machine-learning to effectively measure the impact of our feature engineering pipelines.