Summary

In this chapter, using a service request forecasting project, we went through a step-by-step process of utilizing big data to serve city governments as well as related civic organizations, from which we processed open data on Apache Spark and then built several models, including regression and time series ARIMA models to predict service demands. With this, we then developed rules for alerts and scores for zip code zone ranking to help cities prepare resources to measure effectiveness and also rank communities.

Specifically, we first selected a supervised machine learning approach with a focus on time series modeling per use case needs after we prepared Spark computing and loaded in preprocessed data. Secondly, we worked on data and feature preparation by merging a few datasets together and selecting a core set of features from hundreds of features. Thirdly, we estimated model coefficients using the Zeppelin notebook with MLlib and the R notebook on Databricks. Next, we evaluated these...