Concrete implementations of the use cases that will be discussed in this chapter can be found at https://github.com/PacktPublishing/Java-Deep-Learning-Cookbook/tree/master/02_Data_Extraction_Transform_and_Loading/sourceCode/cookbook-app/src/main/java/com/javadeeplearningcookbook/app.

After cloning our GitHub repository, navigate to the Java-Deep-Learning-Cookbook/02_Data_Extraction_Transform_and_Loading/sourceCode directory. Then, import the cookbook-app project as a Maven project by importing the pom.xml file inside the cookbook-app directory.

The datasets that are required for this chapter are located in the Chapter02 root directory (Java-Deep-Learning-Cookbook/02_Data_Extraction_Transform_and_Loading/). You may keep it in a different location, for example, your local directory, and then refer to it in the source code accordingly.

ETL is an important stage in neural network training since it involves data. Data extraction, transformation, and loading needs to be addressed before we proceed with neural network design. Bad data is a much worse situation than a less efficient neural network. We need to have a basic understanding of the following aspects as well:

• The type of data you are trying to process
• File-handling strategies

In this recipe, we will demonstrate how to read and iterate data using DataVec.

As a prerequisite, make sure that the required Maven dependencies have been added for DataVec in your pom.xml file, as we mentioned in previous chapter, Configuring Maven for DL4J recipe.

The following...

Data transformation is an important data normalization process. It's a possibility that bad data occurs, such as duplicates, missing values, non-numeric features, and so on. We need to normalize them by applying schema transformation so that data can be processed in a neural network. A neural network can only process numeric features. In this recipe, we will demonstrate the schema creation process.

1. Identify the outliers in the data: For a small dataset with just a few features, we can spot outliers/noise via manual inspection. For a dataset with a large number of features, we can perform Principal Component Analysis (PCA), as shown in the following code:

INDArray factor...

The next step after schema creation is to define a data transformation process by adding all the required transformations. We can manage an ordered list of transformations using TransformProcess. During the schema creation process, we only defined a structure for the data with all its existing features and didn't really perform transformation. Let's look at how we can transform the features in the datasets from a non-numeric format into a numeric format. Neural networks cannot understand raw data unless it is mapped to numeric vectors. In this recipe, we will build a transformation process from the given schema.

1. Add a list of transformations to TransformProcess....

DataVec gives us the ability to serialize the transforms so that they're portable for production environments. In this recipe, we will serialize the transformation process.

1. Serialize the transforms into a human-readable format. We can transform to JSON using TransformProcess as follows:

String serializedTransformString = transformProcess.toJson()

We can transform to YAML using TransformProcess as follows:

String serializedTransformString = transformProcess.toYaml()

After the transformation process has been defined, we can execute it in a controlled pipeline. It can be executed using batch processing, or we can distribute the effort to a Spark cluster. Previously, we look at TransformProcessRecordReader, which automatically does the transformation in the background. We cannot feed and execute the data if the dataset is huge. Effort can be distributed to a Spark cluster for a larger dataset. You can also perform regular local execution. In this recipe, we will discuss how to execute a transform process locally as well as remotely.

1. Load the dataset into RecordReader. Load the CSV data in the case of CSVRecordReader:

RecordReader reader = new...

Normalization makes a neural network's job much easier. It helps the neural network treat all the features the same, irrespective of their range of values. The main goal of normalization is to arrange the numeric values in a dataset on a common scale without actually disturbing the difference in the range of values. Not all datasets require a normalization strategy, but if they do have different numeric ranges, then it is a crucial step to perform normalization on the data. Normalization has a direct impact on the stability/accuracy of the model. ND4J has various preprocessors to handle normalization. In this recipe, we will normalize the data.

1. Create a dataset...