Deploying Spark on a cluster with YARN

Yet Another Resource Negotiator (YARN) is Hadoop's compute framework that runs on top of HDFS, which is Hadoop's storage layer.

YARN follows the master-slave architecture. The master daemon is called ResourceManager and the slave daemon is called NodeManager. Besides this application, life cycle management is done by ApplicationMaster, which can be spawned on any slave node and would be alive during the lifetime of an application.

When Spark is run on YARN, ResourceManager performs the role of the Spark master and NodeManagers works as executor nodes.

While running Spark with YARN, each Spark executor is run as a YARN container.

Getting ready

Running Spark on YARN requires a binary distribution of Spark that has YARN support. In both the Spark installation recipes, we have taken care of this.

How to do it...

1. To run Spark on YARN, the first step is to set the configuration:

        HADOOP_CONF_DIR: to write to HDFS
YARN_CONF_DIR: to connect to YARN ResourceManager
$ cd /opt/infoobjects/spark/conf (or /etc/spark)
$ sudo vi spark-env.sh
export HADOOP_CONF_DIR=/opt/infoobjects/hadoop/etc/Hadoop
export YARN_CONF_DIR=/opt/infoobjects/hadoop/etc/hadoop

• You can see this in the following screenshot:

2. The following command launches YARN Spark in the yarn-client mode:

        $ spark-submit --class path.to.your.Class --master yarn --deploy-mode client 
          [options] <app jar> [app options]

Here's an example:

        $ spark-submit --class com.infoobjects.TwitterFireHose --master yarn --deploy-
          mode client --num-executors 3 --driver-memory 4g --executor-memory 2g --
            executor-cores 1 target/sparkio.jar 10

3. The following command launches Spark shell in the yarn-client mode:

        $ spark-shell --master yarn --deploy-mode client 

4. The command to launch the spark application in the yarn-cluster mode is as follows:

        $ spark-submit --class path.to.your.Class --master yarn --deploy-mode cluster 
          [options] <app jar> [app options]

Here's an example:

        $ spark-submit --class com.infoobjects.TwitterFireHose --master yarn --deploy-
          mode cluster --num-executors 3 --driver-memory 4g --executor-memory 2g --
            executor-cores 1 target/sparkio.jar 10

How it works...

Spark applications on YARN run in two modes:

• yarn-client: Spark Driver runs in the client process outside of the YARN cluster, and ApplicationMaster is only used to negotiate the resources from ResourceManager.
• yarn-cluster: Spark Driver runs in ApplicationMaster, spawned by NodeManager on a slave node.

The yarn-cluster mode is recommended for production deployments, while the yarn-client mode is good for development and debugging, where you would like to see the immediate output. There is no need to specify the Spark master in either mode as it's picked from the Hadoop configuration, and the master parameter is either yarn-client or yarn-cluster.

The following figure shows how Spark is run with YARN in the client mode:

The following figure shows how Spark is run with YARN in the cluster mode:

In the YARN mode, the following configuration parameters can be set:

• --num-executors: To configure how many executors will be allocated
• --executor-memory: RAM per executor
• --executor-cores: CPU cores per executor