Memory settings and garbage collection for Spark

Memory management is very crucial for distributed training with large datasets in production. It directly influences the resource consumption and performance of the neural network. Memory management involves configuring off-heap and on-heap memory spaces. DL4J/ND4J-specific memory configuration will be discussed in detail in Chapter 12, Benchmarking and Neural Network Optimization.

In this recipe, we will focus on memory configuration in the context of Spark.

How to do it...

1. Add the --executor-memory command-line argument while submitting a job to spark-submit to set on-heap memory for the worker node. For example, we could use --executor-memory 4g to allocate 4 GB of memory...