Chapter 4: Cleansing, Transforming, and Shaping Data

For data to be used effectively in any kind of reporting, analytics, or AI use case, it must be clean and ready to be joined or shaped with other data. When data is viewed only in the context of the source system that creates it, we're often limited in how we can use it.

For example, if we have sales data coming from point-of-sale terminals, then we can draw conclusions about the total amount of sales completed on any given day, week, or month by simply summing the sales for a given time period. However, if we could join the sales data with weather data, then we could possibly draw conclusions about the impact weather has on sales. Perhaps we believe that rainy weather will have a negative impact on the sales for a given location. In order to test this hypothesis by correlating sales data with weather data, we'll need to ensure things such as that the date and time fields in the weather data can be joined with the date...