Data and feature preparation

In the Feature extraction section of Chapter 2, Data Preparation for Spark ML, we reviewed a few methods of feature extraction and discussed their implementation in Apache Spark. All the techniques discussed there can be applied to our data here.

Besides feature development, for this project, we will also need to spend a lot of effort in merging various datasets together to obtain more features.

Therefore, for this project, we actually need to conduct feature development, then data merging, and then feature selection, which is to utilize all the techniques discussed in Chapter 2, Data Preparation for Spark ML and Chapter 3, A Holistic View on Spark.

Data merging

To obtain features for predicting, we need to add some external datasets, including weather data from National Weather Service Forecast Office, events as well as calendar data from the Open Data portal, and socio-economic data for each zip code block from census data source.

In the, Joining data section of...