Data wrangling with Apache Spark and MLlib

Data wrangling, also referred to within the data science community as data munging, or simply data preparation, is the first step in a typical data science process. Data wrangling involves sampling, exploring, selecting, manipulating, and cleansing data to make it ready for ML applications. Data wrangling takes up to 60 to 80 percent of the whole data science process and is the most crucial step in guaranteeing the accuracy of the ML model being built. The following sections explore the data wrangling process using Apache Spark and MLlib.

Data preprocessing

Data preprocessing is the first step in the data wrangling process and involves gathering, exploring, and selecting the data elements useful for solving the problem at hand. The data science process typically succeeds the data engineering process and the assumption here is that clean and integrated data is already available in the data lake. However, data that is clean enough for...