Implementation and deployment

Implementation depends on setting up the big data infrastructure. Please verify that your MongoDB installation is running properly. Now we shall list implementation objectives as follows:

• Splitting data into test, train and validation datasets
• Data ingestion
• Data analysis

 

Implementation objectives

The overall objective is to perform data analysis on an on-time flight dataset corresponding to the year 2007-2008. Of the 2007 flight data, 80% will be used as the training dataset and the rest as a validation dataset. In so far as model performance evaluation is concerned, 100% of the 2008 flight data becomes the testing dataset.

The following are the implementation objectives required to implement the flight prediction model:

• Download the flight dataset.
•  You may develop the pipeline in four ways:
• Incrementally in your local Spark shell
• By firing up your Horton Sandbox on your host machine managed virtual machine, and developing code in a powerful Zeppelin Notebook environment...