Running the Spark Job on YARN (local)
Hadoop has a long history, and in most cases, organizations have already invested in the Hadoop infrastructure before they move their MR jobs to Spark. Unlike the Spark standalone cluster manager, which can run only Spark jobs, and Mesos, which can run a variety of applications, YARN runs Hadoop jobs as first-class. At the same time, it can run Spark jobs as well. This means that when a team decides to replace some of their MR jobs with Spark jobs, they can use the same cluster manager to run Spark jobs. In this recipe, we'll see how to deploy our Spark application on the YARN cluster manager.
How to do it...
Running a Spark job on YARN is very similar to running it against a Spark standalone cluster. It involves the following steps:
Installing the Hadoop cluster.
Starting HDFS and YARN.
Pushing the Spark assembly and dataset to HDFS.
Running the Spark Job in the
yarn-clientmode.Running the Spark Job in the
yarn-clustermode.
