Running Spark in YARN
In the previous section, we covered Spark working in a Standalone cluster, and while on-premises deployment on Standalone clusters accounts for 42% of all Spark deployments, Spark deployment on YARN accounts for 36% of all on-premises deployments. These figures were taken from the Apache Spark survey report published by data bricks in June of 2016.
Figure 8.15: YARN deployments - Databricks survey report: Page 10 (Databricks.com)
YARN has been supported since 0.6 release of Apache Spark. In order to run your application on YARN you need to ensure you have the correct configuration files for your Hadoop cluster, and the following environment variables point towards the right location of these files:
HADOOP_CONF_DIRYARN_CONF_DIR
These configuration files will be shipped to the YARN cluster and then distributed to all the containers running the application to make sure that they use same configuration.
When you run an application on YARN you have two deployment modes each...
