Configuring job queues

In this recipe, we will configure the job queue and allow users to submit jobs to the queues. In production, there might be many departments such as marketing, sales, and finance, sharing a cluster of resources and it is important to have the correct shares proportional to business and funding.

In the previous recipe, although the queues were setup, they were still not used. Queues were dynamically created for jobs submitted by specifying a queue. If no queue is specified, jobs are submitted to a queue by the name of the user who submitted the job. We will explore these a bit more in this recipe.

Getting ready

To complete the recipe, the user must have a running cluster with HDFS and YARN configured and must have completed the previous two recipes.

How to do it...

1. Connect to the master1.cyrus.com master node in the cluster and switch as user hadoop.

2. Edit the fair-scheduler.xml allocation file as shown next. Note that there is no user specified for any queue (within the ...