Data exploration

Now that we have a clean data frame with relevant data and the initial hypothesis, it is time to really explore what we have. The DataFrames abstraction provides functions such as group by out of the box for you to look around. You may register the cleaned data frame as a table and run the time-tested SQL statements to do just the same.

This is also the time to plot a few graphs. This phase of visualization is the exploratory analysis mentioned in the data visualization chapter. The objectives of this exploration are greatly influenced by the initial information you garner from the business stakeholders and the hypothesis. In other words, your discussions with the stakeholders help you know what to look for.

There are some general guidelines that are applicable for almost all data science assignments, but again subjective to different use cases. Let us look at some generic ones:

• Look for missing data and treat it. We have already discussed various ways to do this in Chapter...