Processing data as a chunk

The XDF file allows for processing tasks more easily by applying the operation chunk by chunk. You do not need to import the entire dataset into the memory to do the processing tasks. To speed up the algorithm or any processing task, the chunk by chunk operation is quite helpful. In this recipe, you will see the chunk by chunk operation.

Getting ready

The XDF file you have created in the previous recipe Creating an XDF file from CSV input in this chapter, contains the following nine variables:

• YEAR
• QUARTER
• MONTH
• DAY_OF_MONTH
• DAY_OF_WEEK
• ORIGIN
• DEST
• DEP_DELAY
• ARR_DELAY

The objective of this recipe is to create a new binary variable binDelay, representing an indicator variable; if the departure delay...